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PREFACE 


The  idea  for  this  conference  arose  early  in  1990  as  the  work  of  revising  the  Department  of 
Defense  5000  series  of  acquisition  regulations  was  underway.  The  impetus  for  the  conference 
was  a  recognition  within  the  Office  of  the  Assistant  Secretary  of  Defense  for  Program  Analysis 
and  Evaluation  (OASD(PA&E))  that  analysis  to  support  acquisition  decisions  plays  a  significant 
role  in  weapon  system  reviews  and  that  the  acquisition  community  would  be  well  served  by  a 
forum  to  explain  and  interpret  the  new  analysis  requirements. 

The  OASD(PA&E)  planned  and  organized  a  series  of  six  Cost  and  Operational 
Effectiveness  Analysis  (COEA)  conferences.  The  first  of  these  occurred  in  April  1991  at  the 
Defense  Systems  Management  College  and  was  attended  by  the  senior  leadership  of  all  the 
military  departments  and  the  Office  of  the  Secretary  of  Defense  (OSD).  At  this  conference  key 
acquisition  analysis  issues  were  defined  and  used  as  the  basis  for  the  agenda  and  topics  presented 
at  later  conferences.  In  May  1991,  three  "military  action  officer"  workshops  were  held  at  the 
MITRE  Corporation.  These  conferences  focused  on  what  constitutes  a  COEA  and  how  it  fits 
into  the  new  DoD  acquisition  management  process.  In  June  1991,  a  COEA  seminar,  opened  to 
the  general  Military  Operations  Research  (MORS)  community,  was  conducted  in  conjunction 
with  the  annual  MORS  meeting  at  the  Naval  Postgraduate  School.  Finally,  in  March  1992,  a 
MORS  sponsored  mini-symposium  was  held.  The  proceedings  herein  record  the  result  of  that 
mini-symposium. 
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CHAPTER  1 
INTRODUCTION 
MORS  MINI-SYMPOSIUM 
COEA  IN  THE  ACQUISITION  PROCESS 
AND  THE  ROLE  OF  OPERATIONS  RESEARCH 
IN  PERFORMING  COEA 


1.0  EXECUTIVE  SUMMARY  The  Military 
Operations  Research  Society  (MORS)  sponsored  a  mini¬ 
symposium  in  Newport,  R1  from  March  9-11,  1 992  on 
"COEA  in  the  Acquisition  Process  and  the  Role  of  OR  in 
Performing  COEA."  Two  hundred  and  forty  participants 
convened  at  the  Newport  Marriot  to  explore  the  COEA 
process  and  to  identify  how  the  application  of  operations 
research  principles  and  techniques  might  improve 
decision  making  in  DoD.  Recurring  themes  at  the 
conference  included  uncertainty  in  the  security 
environment  and  the  problems  it  generates,  turbulence  in 
the  OR  community  as  it  organizes  and  prepares  for  a 
potentially  new  role,  and  recognition  that  the  community 
must  arrive  at  a  consensus  on  its  future  direction. 

The  mini-symposium  was  the  last  in  a  series  of 
COEA  workshops  sponsored  by  the  ASD(PA&E)  to 
assist  operations  researchers  to  transition  to  new  analysis 
requirements  promulgated  by  the  revised  DoD 
acquisition  regulations.  Previous  workshops  focused  on 
explanations  of  new  policies.  The  mini-symposium  was 
aimed  at  initiating  a  dialogue  among  policymakers  and 
practitioners  on  implementing  the  regulations. 

The  objectives  of  the  conference  were  to  promote 
understanding  of  the  role  of  COEAs  in  the  DoD  5000 
Series  acquisition  regulations,  to  define  the  role  of  OR  in 
the  COEA  process,  to  identity  challenges  and  issues 
associated  with  conducting  COEAs,  to  develop  possible 
solutions  or  research  initiatives  for  resolving  issues,  and 
to  improve  the  collaborative  framework  for  dealing  with 
the  COEA  requirements. 

The  three-day  mini-symposium  began  with  a 
keynote  address  by  Dr  David  Chu,  ASDfPA&E).  In  his 
address.  Dr  Chu  emphasized  the  growing  importance  of 
COEAs  in  supporting  and  documenting  acquisition 
decisions.  He  noted  that  the  community  has  a 
responsibility  to  ensure  its  collective  work  is  consistent, 
credible,  and  independent.  He  also  stressed  the  need  to 
correlate  COEAs  with  other  supporting  analyses  and 
documents. 


Representative  Ron  Machtley  (R-RI),  a  member 
of  the  House  Armed  Services  Committee,  addressed  the 
conference  on  congressional  activities  driving  toward 
downsizing  of  military  forces  and  exploiting  the  peace 
dividend.  He  provided  excellent  insights  into  current 
defense  budget  debates  and  discussed  the  general 
economic  impact  of  going  below  current  proposed  budget 
levels.  He  stressed  that  a  steeper  defense  drawdown 
would  require  very  large  additional  reductions  in 
manpower.  Mr  Machtley  also  sketched  out  a  plan,  to 
which  he  assigned  a  high  congressional  priority,  for 
converting  business  from  defense  to  commercial 
enterprise. 

The  mini-symposium  included  presentations  by 
subject  matter  experts  followed  by  question  and  answer 
sessions.  The  speakers  and  subjects  for  this  meeting 
included: 

1)  Dr  Bill  Lese  (OASD(PA&E))  -  Milestone 
Decision  Criteria 

2)  LtCol  Gary  Fauss  (DIA)  -  Threats  and 
Scenarios 

3)  Mr  Sam  Gardiner  -  Modeling  and  Tools 

4)  Mr  Tony  Brinkley  (Teledyne  Brown)  - 
Operational  Effectiveness  Methodologies 

5)  Dr  Dan  Nussbaum  (CNCA)  -  Cost 
Methodologies 

6)  Dr  Pat  Cassady  (TRAC-WSMR)  -  Cost  and 
Effectiveness  Integration 

7)  LtCol  Jim  Feigley  (USMC)  -  Managing 
COEA 

8)  Dr  Bill  Lese  (PA&E),  Dr  Ernie  Seglie 
(DOTE),  and  Dick  Ledesma  (DDR&E)  - 
Linkage  of  Testing  and  COEA 

9)  Dr  A1  Diaz  (OASD(PA&E))  -  Coea 
Guidelines. 

The  mini-symposium  also  included  six  working 
groups,  each  chartered  to  consider  and  develop  issues  on 
various  aspects  (i.e.  functional  areas)  of  the  COEA 
process.  These  groups  were  constituted  from  attendees 
pre-assigned  by  expression  of  interest  or  expertise  in  the 
functional  area.  The  group  chairmen  and  topics  were: 
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1)  Mr  John  Gehrig  (ODUSA(OR)  -  Test  and 
Evaluation 

2)  Col  Charlie  Cox  (DIA)  -  Threat 
Assessments 

3)  Dr  John  Friel  (RAND)  -  Effectiveness 

Methodologies 

4)  Dr  Dan  Nussbaum  (CNCA)  -  Cost 

Methodologies 

5  Col  Don  Bourdon  (NWC)  -  COEA 
M  . .  /  ament 

6)  i'.-jir  Mike  Bauman  (TRAC)  -  Future 
Direction. 


A  "synthesis"  working  group,  chaired  by  Mr 
Clayton  Thomas  (USAFSA),  was  also  functioning  during 
the  meeting.  This  group  consisted  of  experienced  MOR 
analysts  whose  purpose  was  to  observe  the  mini- 
symposium  proceedings  and  to  identify,  synopsize,  and 

present  issues  meriting  further  mvestigation  m  future 
MORS  meetings. 

Finally,  as  key  element  of  the  mini-symposium,  a 
panel  of  "Senior  DoD  Officials"  discussed  their  views  on 
the  future  direction  of  analysis.  The  panel  included  Dr 
Herb  Puscheck  (OASD(PA&E)),  Mr  Walt  Hollis 
(DUSA-OR),  MG  Joe  Ralston  (USAF),  RADM  R.  C. 
Allen  (USN),  and  Mr  Fred  Bielan  (HQ  USMC).  The 
panel  was  moderated  by  Dr  Bill  Lese  (OASD(PA&E)). 
Each  member  of  the  panel  presented  a  "Service" 
perspective  on  analysis  issues  and  then  fielded  questions 
from  the  conferees. 
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1.2  General  Chairman's  Statement 

Good  morning  ladies  and  gentlemen.  As  the 
general  chairman  for  this  meeting,  I  welcome  all  of  you  to 
Newport  and  to  this  MORS  Mini-Symposium  on 
COEAs.  It  is  indeed  a  pleasure  to  see  so  many  of  you 
here  because  it  is  evidence  of  the  importance  that  you 
attach  to  this  topic  and  the  work  associated  with  it.  I 
assure  you  that  this  topic  is  also  of  considerable 
importance  in  the  Department  of  Defense.  And,  as  active 
participants  in  the  community'  of  defense  analysts,  it  is 
good  that  we  take  this  opportunity  to  come  together  to 
discuss  common  problems,  issues,  and  future  directions 
for  our  business. 

While  it  is  true  that  the  subject  of  COEAs  is  not 
new,  there  are  many  reasons  why  it  is  still  a  topic  worthy 
of  this  assembly.  Let  me  point  out  just  a  couple.  First, 
the  national  security  environment  has  changed.  I  will  not 
belabor  the  point  of  what  these  changes  are;  most  of  you 
know  them  well.  But  with  that  change  has  come  a  need 
to  consider  how  analysis,  in  general,  and  COEAs,  in 
particular,  also  have  to  adjust.  You  here  today  are  the 
leaders,  teachers,  and  pathfinders,  in  government  and 
industry,  for  identifying  how  analysis  can  make  the 
transition.  As  a  community,  you  must  take  that  charge 
seriously,  because  if  analyses,  or  analysts,  do  not  support 
the  new  decision  process,  it  and  we  both  will  be  found 
dispensable. 

Second,  I  submit  to  you  that  the  nature  of  the 
requirement  for  analysis  has  changed.  I  do  not  pretend  to 
know  exactly  how  it  has  changed.  But  I  would  argue,  that 
in  the  past,  our  community  was 


focused  on  model  building  and  the  application  of 
modelling  methodologies  in  very  standard  types  of 
problems,  such  as  comparing  weapon  system  alternatives 
using  standard  scenarios.  Today,  I  believe,  we  are 
expected  to  focus  on  problem  formulation  and  responsive 
problem  solving  for  a  broader  set  of  defense  issues  such 
as  force  planning  and  design,  affordability,  and  functional 
complementarity  and  redundancy  of  forces.  The  issues  are 
changing  too  fast  for  old  techniques  to  be  supportive 
either  in  time  or  in  scope.  We  have  to  find  new 
approaches  for  today's  problems. 

The  agenda  we  have  for  this  meeting  is  a 
challenging  one.  We  actually  have  two  meetings  in  one. 

The  first  is  designed  to  communicate  information 
to  you  by  subject  matter  experts  who  hopefully  will 
provoke  your  thinking  and  prove  to  be  a  good  resource 
for  you  in  your  working  groups.  The  second  meeting  is 
designed  to  elicit  from  you  what  you  know.  Your  active 
participation  in  your  working  group  is  key  for  this 
purpose.  I  urge  you  to  be  involved.  Contribute  to  your 
working  groups.  Make  it  a  personal  goal  of  yours  that 
your  contribution  and  your  views  will  be  reflected  in  the 
proceedings.  If  everyone  does  that,  we  will  assure 
ourselves  a  very  productive  meeting. 

-  Alfonso  A.  Diaz 

General  Chairman 
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1.3  Host's  Welcome  Address 


I  welcome  you  to  Newport.  The  Naval  War 

College  is  pleased  to  host  this  MORS  COEA  mini- 

Symposium.  I  believe  that,  in  this  era  of  declining 
budgets,  this  is  a  topic  of  particular  importance  to  the 
Defense  analysis  community.  As  you,  no  doubt,  are 
acutely  aware,  the  advent  of  tighter  DoD  budgets  has 
brought  with  it  the  need  for  closer  scrutiny  of 
expenditures  of  taxpayers  dollars.  DoD,  with  Dr  Chu's 
organization,  PA&E,  in  the  lead,  wants  the  analysis 
community  to  contribute  heavily  to  this  process. 
Therefore,  it  has  been  seeking  to  institutionalize  COEAs 
as  an  integral  part  of  the  DoD  decision  process  leading  to 
defense  expenditures. 

COEAs  play  a  central  role  by  helping  to  determine 
whether  a  system  can  provide  needed  capabilities  at  an 
acceptable  price.  They  are  intended  to  aid  decision 
makers  in  dealing  with  complexities  and  uncertainties  in 
weapons  acquisitions.  They  are  also  intended  to  evaluate 
the  cost  and  benefits  of  alternative  courses  of  action  that 
could  be  taken  to  meet  recognized  defense  needs.  They 
provide  information  on  the  sensitivity  of  acquisition 
alternatives  to  potential  changes  in  key  assumptions. 


variables  and  constraints,  of  both  estimated  costs  and 

estimated  benefits.  As  such,  they  provide  critical  inputs 

to  decisions  on  major  defense  acquisitions  and  a  sound 
analytical  basis  for  evaluating  decision  alternatives. 

This  conference,  the  fourth  in  a  series  on  this 
topic,  but  the  first  one  open  to  the  general  defense 
analysis  community,  is  being  sponsored  to  help  answer 
commonly  asked  questions  about  COEAs  such  as:  How 
are  they  done?  How  do  they  play  in  the  decision  process? 
How  should  they  be  structured  to  best  support 
decisionmakers?  These  are  important  questions  that  you 
as  analysts  and  implementers  of  the  new  acquisition 
process  must  lead  the  rest  of  the  defense  community  in 
answering.  Your  understanding  of  these  issues  will  lead 
to  better  decisions  and  greater  confidence  that  US 
security  interests  are  being  met  effectively  and 
economically. 

Once  again,  welcome,  and  I  wish  you  success  as 
you  tackle  these  important  issues. 

-  RADM  Joesph  Strasser 
President,  Naval  War  College 
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CHAPTER  2 

INVITED  PRESENTATIONS 
MORS  COEA  SYMPOSIUM 
GENERAL  SESSION  TOPIC 


2.0  Abstracts 

Analysis  Requirements  For  Milestone  Review: 

The  acquisition  of  military  weaponry  and 
equipment  is  one  of  the  basic  functions  of  the 
Department  of  Defense  (DoD).  Accordingly,  oversight 
and  review  of  major  acquisition  programs  has  become 
a  high  management  priority  for  the  department's  senior 
leadership.  As  new  systems  move  from  the  drawing 
board  through  development  and  production,  they  are 
reviewed  at  regular  intervals  by  decisionmakers  within 
both  the  military  departments  and  the  Office  of  the 
Secretary  of  Defense  (OSD).  The  purpose  of  these 
reviews  is  to  assess  the  performance  potential  and  costs 
of  the  systems,  so  as  to  determine  whether  they  can 
provide  needed  capabilities  at  an  acceptable  price.  Cost 
and  operational  effectiveness  analyses,  or  COEAs  as 
they  are  commonly  called,  play  a  central  role  in  the 
review  process. 

COEAs  are  intended  to  evaluate  the  costs  and 
benefits  -  operational  effectiveness  -  of  alternative 
courses  of  action  that  could  be  taken  to  meet  a 
recognized  defense  need.  They  also  should  provide 
information  on  the  sensitivity  of  acquisition  alternatives 
to  potential  changes  in  key  assumptions,  variables,  and 
constraints  of  both  the  estimated  costs  and  the  estimated 
benefits.  Thus,  they  provide  critical  inputs  to  decisions 
on  major  defense  acquisitions. 

This  presentation  will  provide  an  overview  of  the 
defense  acquisition  process  and  the  role  COEAs  play  in 
that  process  and  will  explain  how  COEA  results  are 
used  by  the  Office  of  the  Secretary  of  Defense. 


Threat  Assessments/STAR  and  Scenarios 

The  Defense  Intelligence  Agency  (DLA) 
provides  threat  support  to  the  Defense  Acquisition 
Board  (DAB),  Joint  Chiefs  of  Staff  (JCS),  and  the 
Services.  The  DoD  5000  Series  regulations  require  that 
DIA  validate  all  threat  information  used  to  support 
DAB  program  decisions.  DIA  also  validates  the 
System  Threat  Assessment  Report  (STAR)  and  the 
threat  portion  of  other  program  documentation, 
including  the  Cost  and  Operational  Effectiveness 
Analysis  (COEA).  Validation  of  these  documents 


ensures  that  the  intelligence  is  complete,  appropriate, 
reasonable,  consistent,  and  logical. 

The  validated  threat  baseline  for  the  COEA  of 
major  programs  depends  on  where  the  program  is  in  the 
acquisition  cycle.  Prior  to  Milestone  I,  a  series  of  threat 
descriptions,  by  mission  areas,  are  used.  For  Milestone 
I,  a  STAR  focused  on  the  threat  environment  expected 
for  the  specific  program  is  prepared  by  the  Service  and 
validated  by  DIA.  The  STAR  is  updated  prior  to  each 
milestone,  or  when  a  significant  change  occurs  in  the 
threat. 

DoD  Instruction  5000.2  states  that  scenarios  for 
COEAs  be  based  on  the  set  of  Defense  Planning 
Guidance  (DPG)  scenarios.  DIA  assists  OSD  and  JCS 
in  the  DPG  process  of  developing  these  scenarios. 
Alternative  scenarios  may  be  considered  in  the  COEA. 
In  either  case,  DIA  must  review  threat  assumptions  and 
data. 

Since  the  promulgation  of  the  DoD  5000  Series 
guidance,  DIA  has  been  more  involved  in  the  COEA 
process.  DIA  support  will  be  ramped-up  as 
requirements  are  levied.  Validation  of  the  threat 
models  used  in  the  COEA  and  requirements  for  more 
threat  detail  than  is  provided  in  the  DPG  and  STAR  are 
areas  of  concerns.  As  a  result,  DIA  needs  to  be 
involved  early  in  the  COEA  planning  process. 
Coordination  can  be  handled  through  the  local 
intelligence  support  organizations,  COEA  steering 
groups,  or  threat  working  groups. 

Modeling  and  Tools 

The  Wall.  The  Gulf  War.  New  Doctrines.  The 
disintegration  of  the  Soviet  Union.  New  weapons 
technologies.  War  and  warfare  will  be  very  different. 
If  analysis  is  going  to  contribute  to  an  understanding  of 
the  future,  it  will  have  to  be  done  in  different  ways.  If 
analysis  is  going  to  contribute  to  the  acquisition 
process,  it  will  have  to  be  done  in  different  ways.  It 
will  have  to  have  a  different  focus.  It  will  have  to  have 
different  components.  This  presentation  will 
characterize  the  changes  in  the  nature  of  war  and 
warfare  and  will  suggest  the  kinds  of  changes  necessary 
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for  productive  analysis  in  the  our  world,  the  kind  of 
analysis  necessary  for  COEAs. 


Operational  Effectiveness  Methodologies 

Within  the  conduct  of  a  Cost  and  Operational 

Effectiveness  Analysis  (COEA),  the  basic  problem  to 
be  dealt  with  is  the  development  and/or  selection  of  an 
appropriate  methodology  to  compare  current  and 
proposed  material  systems  within  an  operational 
context.  Operational  effectiveness  (the  operational 
context)  should  not  automatically  imply  a  specific  force 
level  (i.e.,  joint/combined,  theater,  corps,  brigade,  etc.) 
at  which  comparative  force-on-force  analysis  must  be 
conducted  to  assess  differences  between  forces 
equipped  with  current  and  future  systems.  Neither 
should  operational  effectiveness  imply  specific  models 
or  types  of  models  to  be  used  to  support  COEAs. 
Rather,  COEA  analytical  methodologies  should  be 
developed  or  selected  through  rigorous,  front-end 
analysis  of  specific  issues  and  questions  that  must  be 
answered  to  provide  senior  decisionmakers  with  the 
best  possible  information  on  which  to  base  "go"  or  "no 
go"  decisions.  This  front-end  analysis  of  issues  is  a 
joint  responsibility  of  the  decisionmakers  and  those 
who  do  COEA.  It  should  lay  out  clearly  the  depth  and 
breadth  constraints  within  which  the  operational 
effectiveness  analysis  will  be  conducted.  Tailored 
analytical  approaches  and  methodologies,  linked  to 
decision  issues  and  to  the  time  available,  are 
significantly  more  useful  and  less  resource  demanding 
than  adherence  to  a  "cookbook"  methodology. 
Likewise,  responsive  and  transparent  analytical  tools 
and  models  should  be  preferred  in  many  COEAs  over 
the  more  complex,  non-transparent  "black  boxes"  so 
often  selected  to  support  these  types  of  studies  and 
analyses. 

Cost  Analysis  in  COEAs 

The  presentation  addresses  four  questions 
associated  with  the  cost  analysis  portion  of  conducting 
a  COEA: 

1)  WHAT  IS  THE  QUESTION?  What  are  the 
terms  for  the  requirement  for  a  cost  analysis  in 
a  COEA? 

2)  WHAT  IS  THE  DELIVERABLE?  What  is 
it  that  the  cost  analyst  owes  to  the  COEA 
process,  and  in  what  format  does  it  get 
presented? 


3)  WHO  IS  DOING  COST  ANALYSIS? 
Professional  communities  are  often  insular. 
The  identification  of  the  cost  analysis 
organizations  involved  in  COEAs  must  also 
show  where  they  fit  in  their  Service 
organizations. 

4)  WHAT  ARE  THE  CURRENT  ISSUES  IN 

COST  ANALYSIS.  While  there  are  a  number 
of  issues  under  intense  debate  within  the  cost 
analysis  community,  several  of  them  earn  the 
spotlight  the  COEA  process: 

-  PRESENT  VALUE  ANALYSIS.  What  is  it? 
Why  do  it?  What  are  the  choices?  What 
research  questions  are  open? 

-  RISK  AND  UNCERTAINTY  ANALYSIS. 
The  description  of  risk  and  uncertainty 
analyses  differentiates  between  the  two  and 
provides  recent,  "live"  examples  of  each. 

-  R&D  COSTING.  An  approach  to  weapon 
system  acquisition  that  has  received  recent 
attention  is  to  do  the  R&D  and  then  to  put  it 
in  the  shelf,  delaying  production  to  some 
indefinite  future.  What  implications  does  this 
delay  have  for  COEAs? 

Cost  and  Effectiveness  Integration 

The  analysts  have  estimated  both  effectiveness 
and  cost  for  each  alternative  considered  in  the  Cost  and 
Operational  Effectiveness  Analysis  (COEA).  Their 
next  task  is  to  rank  order  the  alternatives.  Ranking  the 
alternatives  requires  that  the  distinct  attributes,  cost  and 
effectiveness,  somehow  be  integrated.  This 
presentation  considers  the  integration  problem  from  the 
viewpoint  of  multi-attribute  or  multiple  criteria  decision 
making.  From  this  viewpoint,  an  individual  decision 
maker's  preferences  and  value  trade-offs  among 
multiple  and  often  conflicting  attributes  can  be 
explicitly  modeled.  This  model  can  then  be  used  to 
rank  the  decision  alternatives.  From  existing  literature, 
several  possible  techniques  are  identified.  Two  of  the 
techniques  are  demonstrated  in  the  context  of  an 
abbreviated  COEA  on  the  TOW  Sight  Improvement 
Program. 

Managing  COEA  Development 

The  author's  recent  experience  in  putting 
together  a  COEA  for  a  major  weapons  system  will 
demonstrate  the  management  process.  Emphasis  is 
placed  on  a  practical  approach  to  understanding  what 
must  be  done,  and  on  how  to  organize  oneself  and 
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others  to  accomplish  it.  Subjects  include:  what  to  do 
when  initially  tasked,  how  to  identify  what  needs  to  be 
done,  what  are  the  most  important  elements  of  a  COEA, 
how  to  organize  senior  leadership  and  working  level 
resources,  and  (most  importantly),  what  are  the  lessons 
learned  from  the  entire  experience.  Principles, 
observations,  and  recommendations  are  made  in  such  a 
way  that  they  can  apply  to  any  program  or  weapon 
system. 

Linking  COE  As  to  OT&E 

Current  acquisition  policy  states  that  the  cost  and 
operational  effectiveness  analyses  and  test  and 
evaluation  are  aids  to  decisionmaking.  The  COEA  aids 
decisionmakers  in  judging  whether  any  of  the  proposed 
alternatives  are  a  cost  effective  approach  to  meeting  an 
operational  requirement.  Test  and  evaluation  aids 
decisionmakers  in  verifying  that  systems  have  attained 
their  technical  performance  specifications  and 
objectives,  are  operationally  effective,  and  are 
operationally  suitable  for  their  intended  use.  The 
policies  require  that  a  linkage  exist  between  COEAs 
and  test  and  evaluation. 

In  order  to  implement  this  guidance  consistently 
throughout  the  acquisition  process,  guidelines  have 
been  developed  for  COEA  to  mclude  a  measures  of 
effectiveness  (MOEs)  that  can  be  tested  and  show  how 
other  COEA  MOEs  can  be  derived  from  testable 
parameters.  In  addition,  sensitivity  analyses  conducted 
as  part  of  a  Milestone  II  COEA  should  identify  any 
critical  sensitivities  of  system  effectiveness  to  the  stated 
test  limitations.  The  Defense  Acquisition  Board  (DAB) 
and  the  Defense  Acquisition  Executive  (DAE)  should 
be  able  to  review  the  COEA  results  using  test  results  to 
reaffirm  the  decision  that  the  selected  alternative 
continues  to  be  a  cost  effective  approach  to  satisfying 
the  operational  requirement. 


In  theory,  linkage  between  the  Mission  Needs 
Statement  (MNS),  Operational  Requirement  Document 
(ORD),  COEA,  Acquisition  Program  Baseline  (APB), 
and  Test  and  Evaluation  Master  Plan  (TEMP)  in 
measures  of  effectiveness,  measures  of  peifonnance, 
and  criteria  seems  obvious.  So  does  ensuring  that  test 
results  are  used  to  "validate"  the  performance 
assumptions  underlying  the  COEA  results  and  that  the 
premises  for  earlier  acquisition  decisions  remain  sound. 
In  practice,  this  linkage  is  difficult  to  achieve  and 
presents  numerous  analytical  challenges.  These 
challenges  will  be  highlighted  by  the  members  of  this 
panel. 

The  COEA  in  Support  of  the  DoD  Decision  Process 

This  presentation  describes  elements  and  criteria 
for  judging  how  well  a  COEA  meets  the  new  DoD 
5000  Series  acquisition  regulations.  It  provides  a 
framework  for  evaluating  a  COEA  that  is  keyed  to  the 
analysis  requirements  incorporated  into  the  regulations. 
Guidelines,  while  not  necessarily  comprising  an 
exhaustive  "checklist,"  are  presented  to  identify  general 
properties  that  characterize  "good"  COEAs.  The 
guidelines  are  offered  as  a  useful  basis  for  achieving 
uniformity  and  consistency  in  doing  COEAs  and  for 
enhancing  the  degree  to  which  COEAs  can  support  the 
DoD  decision  process.  While  the  guidelines  are  offered 
primarily  as  an  aid  for  implementing  the  acquisition 
regulations,  it  is  hoped  that  they  also  will  serve  as  a 
starting  point  for  the  analysis  community  to  take  up  the 
greater  challenge  of  defining  what  constitutes  "high 
quality"  COEAs. 
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PAPERS  AND/OR 
VIEWGRAPHS 
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2.1.1  Cost  and  Operational  Effectiveness  Analysis 


MORS  COEA 
MINI-SYMPOSIUM 
MARCH  9-11.  1992 


COST  AND  OPERATIONAL 
EFFECTIVENESS  ANALYSIS 
(COEA) 


KEYNOTE  ADDRESS 
DR  DAVID  S.  C.  CHU 
ASD(PA&E) 


PURPOSE  OF  THE 
COEA  MINI-SYMPOSIUM 


.  CLARIFY  WHAT  IS  EXPECTED  IN  COEAs 

•  IDENTIFY  AND  DISCUSS  PROBLEMS  OF 
IMPLEMENTATION 

.  IMPROVE  COLLABORATION  IN  SUPPORT 
OF  ACQUISITION  DECISIONS 


PURPOSE  AND  ROLES  OF  COLA 


COEAs  SHOULD: 


FACILITATE  COMMUNICATIONS: 

HELP  IDENTIFY  ALTERNATIVES  EARLY 
MAKE;  ASSUMPTIONS  AND  VARIABLES  EXPLICI  T 


\||)  DECISIONMAKING 

EXAMINE  STRENGTH  &  WEAKNESSES  <  )E  PROG  KA  1 1<  )NAI  I 
ILLUMINATE  PROs  &  CONs  OF  ALTERNATIVES 
MAKE  SENSITIVITIES  EXPLICIT 


DOCUMENT  DECISIONS 

RECORD  LOGIC  &  ANAL  CONSIDERED  BY  DECISIONMAKERS 
PROVIDE  BASIS  FOR  PROGRAM  RATIONAL.F. 


PURPOSE  AND  ROLES  OF  COEA 


# 


NO  SINGLE  "COOKBOOK"  SOLUTION 


CAPABILITY  COUNTS 


# 


•  DO  NOT  FORCE  EQUAL  COST  OR 

EQUAL  EFFECTIVENESS  COMPARISONS 
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PROBLEMS  AND  CONCERNS 

(1) 

THE  RATIONALE  FOR  A  SYSTEM  IS  OFTEN  NOT 
CLEARLY  ESTABLISHED 


•  MISSION  AREA  ANALYSES 

NOT  DONE  OR  NOT  AVAILABLE: 

•  CAMPAIGN  ANALYSES 

NOT  DONE  OR  NOT  AVAILABLE. 

YET  THEY . 

I IEI .1*  ENSURE  CONSISTENCY 
FOCUS  ON  OUTCOMES  AND  KEY 
CHARACTERISTICS 

•  SCENARIOS  AND  THREATS  NOT  KEPT  CURRENT 


PROBLEMS  AND  CONCERNS 

(2) 


ALTERNATIVES 

TOO  NARROW  -  MINOR  VARIATIONS  ON  A  THEME 
NOT  DIRECTLY  LINKED  TO  THE  DECISION 

REQUIREMENTS  TRADE-OFFS 
NOT  VISIBLE 

COMBAT  VALUE  NOT  SHOWN 

MEASURES  OF  EFFECTIVENESS 

SHOW  PERFORMANCE  BUT  NOT  VALUE  TO 
COMBAT  OUTCOME 
NOT  LINKED  TO  BASELINE  THRESHOLD 
DO  NOT  SHOW  IF  SYSTEM  AS  TESTED  IS 
COST-EFFECTIVE 


12 


PROBLEMS  AND  CONCERNS 

(3) 


{•  MODELS  NOT  ALWAYS  CREDIBLE... 

NOR  CAN  RESULTS  ALWAYS  BE  REPLICATED 

|  •  DATA  NOT  CONSISTENT  WITH  DATA  USED  IN 
OTHER  ANALYSES 

j  •  COSTS 

NOT  CONSISTENT  WITH  FIGURES  BRIEFED 
TO  THE  CAIG 

NOT  INCLUSIVE  TO  ALL  COSTS  INCURRED 


PROBLEMS  AND  CONCERNS 

(4) 


•  MANAGING  COEAs 

INDEPENDENCE  IS  ESSENTIAL 

OSD  SHOULD  BE  INVOLVED  THROUGHOUT... 
NO  SURPRISES 

COEA  AND  SSEB  CONCLUSIONS  NOT  ALWAYS 
CONSISTENT 
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HOW  CAN  OPERATIONS  RESEARCH  HELP 
SOLVE  THESE  PROBLEMS  ? 


I  ©FOCUS  ON  MILITARY  OUTCOMES 
|  •SPECIFY  A  BROAD  RANGE  OF  ALTERNATIVES 
•CHALLENGE  "REQUIREMENTS" 

•SCRUTINIZE  SCENARIOS 
•ENFORCE  CONSISTENCY 
4BE  REALISTIC  ABOUT  COSTS 


CONCLUSION 


COEA 


A  GUIDE  TO  DoD  DECISION 


2.1.2  Analysis  Requirements  for  Milestone  Reviews 


MORS  COEA  SYMPOSIUM 


ANALYSIS  REQUIREMENTS 

FOR 

MILESTONE  REVIEWS 


Dr  Bill  Lese 
OASD(PA&E) 


ACQUISITION  MILESTONES  &  PHASES 


DETERMINATION  | 
I  OF  MISSION  . 


PRODUCTION 

( 

DEVELOPMENT 


OPERATIONS 

I 

\  SUPPORT 


CONCEPT  \  I  CONCEPT  H  «*JOR 

STUDE3  DEMONSTATION  I  \  DEVELOPMENT  /  l  PRODUCTION  I  l  MODIFICATION 

APPROVAL  /  \  APPROVAL  /  \  APPROVAL  /  \  APPROVAL  /  \  APPROVAL 


JROC  LMd 
AcquWHon  Support 


AcquMionlud 
JROC  Support 


Logbtlca  Letrt 

AcquMlon  Support 
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MISSION  NEED  DETERMINATION 

MINIMUM  REQUIREMENTS 


OBJECTIVE: 

•  IDENTIFY  DEFICIENCIES  AND/OR  OPPORTUNITIES 

ASSESSMENTS: 

•  THREAT 

•  MISSION  NEED  ANALYSIS 

(DEFINES  DEFICIENCIES/OPPORTUNITIES) 

•  NON  MATERIAL  SOLUTION 

(EVALUATES  DOCTRINE,  OPERATIONAL  CONCEPTS 
TACTICS,  TRAINING  &  ORGANIZATION) 

•  MODS  TO  CURRENT  US/ALLIED  SYSTEMS 

DOCUMENTATION 

•  MISSION  NEED  STATEMENT 

JROC  VALIDATED 
JOINT  PRIORITY  ASSIGNED 


MILESTONE  0  REVIEW 

C(  )N( T.PT  STUDUIES  APPROVAL 

DAB  DECISION  ISSUE 

•  IS  THE  MNS  ACCEPTABLE  BASED  ON 

*  A  VALIDATED  THREAT 

*  CONFIRMATION  OF  MATERIEL  SOLUTION  RQMT 

*  MSN  NEED  BEING  IMPORTANT  ENOUGH  TO  STUDY 

ANALYSIS  NEEDS: 

•  THEATER  LEVEL/  CAMPAIGN  ANALYSIS 

DAB  RESULTS:  AN  ADM  THAT  SPECIFIES 

•  THE  MIN  SET  OF  MATERIEL  ALTERNATIVE  CONCEPTS 

•  WHAT  STUDIES  WILL  BE  AUTHORIZED 

•  DOLLARS  AND  FUNDING  SOURCES  FOR  STUDIES 
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THEATER  LEVEL/CAMPAIGN  ANALYSIS 

CHARACTERISTICS 


i  •  EVALUATES  CROSS-SERVICE  AND  WITHIN 

SERVICE  ALTERNATIVE  CONCEPTS  TO  SATISFY 
IDENTIFIED  DEFICIENCIES  OR  TO  CAPATILIZF 
I  ON  OPPORTUNITIES 


•  CONDUCTED  IN  CONTEXT  OF  JOINT  OPERATIONS 
IN  LARGF  SCALE  SCENARIOS. 


CAPTURES  THE  SYNERGISTIC  EFFECTS  OF  CROSS¬ 
SERVICE  SYSTEMS  OVER  EXTENDED  PERIODS  OF 
CONFLICT 

DEFINES  THE  CHARACTERISTICS  REQUIRED  OF  A 
SYSTEM  TO  MEET  IDENTIFIED  DEFICIENCIES 


MILESTONE  I  REVIEW 

CONCI-PT  DEMONSTRATION  APPROVAI. 


DAB  DECISION  ISSUES: 

•  IS  A  NEW  PROGRAM  START  WARRANTED  BASED  ON 

*  A  VALIDATED  THREAT 

*  CONFIRMATION  OF  NEED  BY  STUDIES 

*  PROGRAM  BEING  AFFORDABLE  VIS-A-VIS 

LONG  RANGE  INVESTMENT  PLANS 
AVAILABILITY  OF  REQUIRED  RESOURCES 
(PEOPLE  AND  FUNDS 

•  IS  CONCEPT  BASELINE  ACCEPTABLE  BASED  ON  SPECIFIED 

*  COST,  SCHEDULE  &  PERFORMANCE  < )BJF.CTIVES 
*ORD  THRESHOLDS 

ANALYSIS  NEEDS: 

•  COEA 

•  THEATER  LEVEL/CAMPAIGN  ANALYSIS 

•  AFFORDABILITY  ANALYSIS 
RESULTS  (DOCUMENTED  IN  THE  ADM) 

•  DEM/VAL  GO-AHEAD 

•  CONCEPT  BASELINE  APPROVAL 
EXIT  CRITERIA  FOR  PHASE  I 
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PHASE  I  DEMONSTRATION  &  VALIDATION 


•  COEA  (MSII  HER  r  ALTERNATIVE  DESIGN  APPROACH) 

•  REVALIDATED  IHREAT 

•  MAJOR  COST,  SCHEDULE  &  PERFORMANCE  TRADE-OFF 
OPPORTUNITIES  DEFINED 

•  DEVELOPMENT  BASELINE  ESTABLISHED 

•  DEVELOPMENTAL  TEST  RESULTS  &  ASSESSMENTS 

•  REFINED  ACQUISITION  STRATEGY 

-  HIGH  RISK  AREAS 

RISK  MANAGEMENT  APPROACH 

-  LRU’  QUANTITIES 

•  ASSESSMENT  OF  INDUSTRIAL  BASE  CAPBILITY 

•  AFFORDABILITY  AND  LCC 

•  ADEQUATE  RESOURCES  PROGRAMMING 

•  PROPOSED  EXIT  CRITERIA  FOR  PHASE  II 

•  POTENTIAL  ENVIRONMENTAL  CONSEQUENCES 


MILESTONE  II  REVIEW 

DEVELOPMENTAL  APPROVAL 


DAB  DECISION  ISSUES: 

•  SHOULD  PROGRAM  PROCEED  TO  EMD  BASED  ON 

4  A  VALIDATED  THREAT 

4  VALIDATED  COST,  SCHEDULE  &  PERF  TRADE-OFFS 
4  PROGRAM  BEING  AFFORDABLE 

•  IS  CONCEPT  BASELINE  ACCEPTABLE  BASED  ON 

-  REFINED  COST,  SCHEDULE  &  PERF  OBJECTIVES 

-  REFINED  ORD  THRESHOLDS 

ANALYSIS  NEEDS: 

•  COEA 

•  STILL  MEETING  ORIGINAL  REQUIREMENT 

•  AFFORDABILITY  ANALYSIS 

DAB  RESULTS  (DOCUMENTED  IN  THE  ADM) 

•  EMD  GO-AHEAD 

•  DEVELOPMENT  BASELINE  APPROVAL 
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PHASE  II  -  EMD 


•  VALIDATED  THREA I 

•  TEST  RESULTS  UNDER  OPERATIONAL  CONDITIONS 

•  LRIP  THAT 

•  VERIFIES  ADEQUACY  OF  PRODUCTION  PROCESS 

•  CONFIRMS  STABILITY  &  PRODUCIBILITY  OF  DESIGN 

•  PROVIDES  REALISTICS  ESTIMATES  OF  PRODUCTION 

COST 

•  REFINr  1 1  ACOUISIT  ION  STRATEGY  &  COST  ESTIMATE 

•  REFINED  PROGRAM  COST.  SCHEDULE'S  PERF  OBJECTIVES 

•  PRODUCTION  BASELINE 

•  SYSTEM  CONFIGURATION  BASELINE 

•  AFFORDABILITY  AND  LCC  ASSESSMENT  UPDATE 

•  SUFFICIENT  PROGRAMMING  RESOURCES 

•  POTENTIAL  ENVIRONMENTAL  CONSEQUENCES 


MILESTONE  III  DECISION  ISSUES 


DAB  DECISION  ISSUES: 

•  SHOULD  PROGRAM  PROCEED  TO  PRODUCTION  BASED  ON 

*  A  VALIDATED  THREAT 

*  VALIDATED  PERFORMANCE  OBJECTIVES  AND  THRESHOLDS 

*  OT  SHOWING  SYSTEM  IS 

STABLE 

-  OPERATIONALLY  ACCEPTABLE 

-  LOGISTICALLY  SUPPORTABLE 

*  LRIP  SHOWING  SYSTEM  CAN  BE  PRODUCED  EFFICIENTLY 

*  PROGRAM  BEING  AFFORDABLE 

•  IS  PRODUCTION  BASE  LINE  ACCEPTABLE  BASED  ON 

REFINED  COST.  SCHEDULE  &  PERF  OBJECTIVES 
REFINED  ORD  THRESHOLDS 

ANALYSIS  NEEDS: 

•  COEA  UPDATE  {IF  THREAT  OR  COST  CHANGES  MUCH) 

•  AFFORDABILITY  ANALYSIS 

DAB  RESULTS  (DOCUMENTED  IN  THE  ADM) 

•  PRODUCTION  GO-AHEAD 

•  PRODUCTION  BASELINE  APPROVAL 

•  EXIT  CRITERIA  FOR  PHASE  III _ 
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PHASE  II  -  EMD 


•  VALIDATED  THREAT 

•  TEST  RESULTS  UNDER  OPERATIONAL  CONDITIONS 

•  LRIP  THAT 

’  VERIFIES  ADEQUACY  OF  PRODUCTION  PROCESS 

•  CONFIRMS  STABILITY  &  PRODUCIBILITY  OF  DESIGN 

•  PROVIDES  RFALISTICS  ESTIMATES  OF  PRODUCTION 

r.osr 

!  *  REFINED  ACQUISITION  STRATEGY  &  COST  ESTIMATE 
•  •  REFINED  PROGRAM  COST,  SCHEDULE  &  PERF  OBJECTIVES 

•  PRODUCTION  BASELINE 

•  SYSTEM  CONFIGURATION  BASELINE 

•  AFFORDABILITY  AND  LCC  ASSESSMENT  UPDATE 

•  SUFFICIENT  PROGRAMMING  RESOURCES 

•  POTENTIAL  ENVIRONMENTAL  CONSEQUENCES 


MILESTONE  III  DECISION  ISSUES 


DAB  DECISION  ISSUES: 

•  SHOULD  PROGRAM  PROCEED  TO  PRODUCTION  BASED  ON 

•  A  VALIDATED  THREAT 

•  VALIDATED  PERFORMANCE  OBJECTIVES  AND  THRESHOLDS 

•  OT  SHOWING  SYSTEM  IS 

STABLE 

-  OPERATIONALLY  ACCEPTABLE 

-  LOGISTICALLY  SUPPORTABLE 

•  LRIP  SHOWING  SYSTEM  CAN  BE  PRODUCED  EFFICIENTLY 

•  PROGRAM  BEING  AFFORDABLE 

•  IS  PRODUCTION  BASE  LINE  ACCEPTABLE  BASED  ON 

REFINED  COST.  SCHEDULE  &  PERF  OBJECTIVES 
REFINED  ORD  THRESHOLDS 

ANALYSIS  NEEDS: 

•  COEA  UPDATE  (IF  THREAT  OR  COST  CHANGES  MUCH) 

•  AFFORDABILITY  ANALYSIS 

DAB  RESULTS  (DOCUMENTED  IN  THE  ADM) 

•  PRODUCTION  GO-AHEAD 

•  PRODUCTION  BASELINE  APPROVAL 

•  EXIT  CRITERIA  FOR  PHASE  III 
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PHASE  IV  -  OPERATIONS 

SUPPORT 


! 

•  VALIDATED  THREAT  ASSESSMENT 

•  UPDATED  CONFIGURATION  BASSLINE 

•  ATTAINMENT/MAINTENANCE  OF  REQUIRED 

PERFORMANCE  CAPABILITIES  (PIPS) 

!  •  CONDUCT  OF  SLEPS  AS  APPROPRIATE 


2.1.3  Threat  Assessments/Star  and  Scenarios 


Overview 

The  threat  assessment  process  for  Cost  and 
Operational  Effectiveness  Analysis  (COEA)  begins 
with  basic  threat  support  and  validation.  Next,  a  threat 
baseline  is  defined  for  the  COEA.  Threat  scenarios  are 
then  identified  which  place  the  threat  baseline  in  the 
proper  context.  Threat  analysis  issues  and  coordination 
will  be  described  and  the  paper  will  conclude  with 
some  recent  experiences  and  concerns. 

Threat  Support  and  Validation 

The  Defense  Intelligence  Agency  (DIA) 
provides  threat  support  to  the  Joint  Chiefs  of  Staff 
(JCS),  the  Office  of  the  Secretary  of  Defense  (OSD), 
Defense  Agencies,  the  Services,  and  also  U&S 
Commands.  For  major  U.S.  weapons  programs,  DIA 
supports  the  Defense  Acquisition  Board  (DAB) 
oversight  process.  Support  includes  providing  threat 
information;  acting  as  a  threat  advisor  to  various  DAB 
committees,  other  OSD  working  groups,  and  the  Joint 
Requirements  Oversight  Council;  and  reviewing  and 
validating  threat  information  developed  by  DoD 
components.  The  DoD  5000  series  regulations 
specifically  require  that  "threat  information,  to  include 
the  target  data  base,  must  be  validated  by  DIA  for 
acquisition  programs  subject  to  review  by  the  DAB..." 

Initial  efforts  begin  with  DIA  input  and  review 
of  the  Defense  Planning  Guidance  (DPG).  DIA  assists 
OSD  and  JCS  in  the  development  of  threat  implications 
in  the  DPG  and  specifically  the  threat  forces  contained 
in  the  Illustrative  Planning  Scenarios  (IPSs)  which  will 
be  described  later.  As  chairman  of  the  Red  Advisory 
Panel,  DIA  also  coordinates  Intelligence  Community 
input,  enemy  order  of  battle,  and  threat  actions  in  the 
JCS  Joint  Military  Net  Assessment  process. 

For  the  acquisition  process,  DIA  reviews  and 
validates  threat  sections  of  the  Mission  Need  Statement, 
Operational  Requirement  Document,  Integrated 
Program  Summary,  Integrated  Program  Assessment, 
Test  and  Evaluation  Master  Plan,  and  COEA.  A  major 
activity  for  a  specific  weapon  program  is  DIA's 
validation  of  the  System  Threat  Assessment  Report 
(STAR),  described  later.  Validation  involves  reviewing 
the  proposed  threat  information  with  the  following 
considerations: 


(1) 

Is  the  full  scope  of  applicable  current 
and  future  threats  considered? 

(2) 

Are  all  threats  shown  appropriate? 

(3) 

Are  the  threats  reasonable,  that  is, 
are  the  treat  systems,  tactics,  and 
force  structure  feasible? 

(4) 

Is  the  threat  information  consistent 
with  the  latest  defense  and  national 
intelligence  position? 

(5) 

Are  extrapolations  of  future  threat 
capabilities  or  technologically 
feasible  options  logical? 

Threat  Baseline 

The  validated  threat  baseline  for  COEAs  will 
be  a  combination  of  the  DPG  and  other  documents 
depending  on  where  the  program  is  in  the  acquisition 
cycle.  The  DPG  contains  the  overall  threat  scenarios. 
The  underlying  assumptions  concerning  the  threat  in 
the  COEA  should  not  conflict  with  the  DPG.  Prior  to 
Milestone  I,  the  initial  threat  will  be  found  in  a  series  of 

baseline  threat  documents  prepared  by  the  Services  and 
validated  by  DIA  or  produced  under  DIA  cover.  These 
threat  documents  are  normally  developed  by  mission 
area.  Examples  include  threats  to  undersea,  airlift, 
space,  and  special  operations  forces.  Both  the  Army's 
Battlefield  Development  Plans  and  Navy's  Pyramid 
publications  are  being  redone  to  reflect  worldwide 
threats.  The  Air  Force  Threat  Environment 
Descriptions  (TEDs)  also  reflect  this  changed  emphasis. 
The  Pyramid  publications  and  TEDs  are  now  produced 
as  formal  DIA  documents.  In  order  to  support  long- 
range  analysis,  these  generic  documents  project  mission 
area  threats  out  20  years. 

For  Milestone  I  and  beyond,  a  STAR  is 
produced.  The  STAR  focuses  on  the  threat  for  a 
specific  weapons  program.  The  STAR  projects  the 
threat  (weapons,  targets,  tactics,  and  order-of-battle)  at 
the  initial  operational  capability  (IOC)  and  again  at  IOC 
+  10  years.  It  includes  likely  reactive  threat  changes 
due  to  premature  disclosure  about  the  program  or  actual 
deployment.  The  STAR  is  updated  prior  to  each 
milestone  or  if  a  significant  change  in  threat  occurs. 
Since  the  COEA  and  STAR  are  being  updated 
simultaneously  before  the  milestone,  it  is  important  that 
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threat  changes  are  carefully  coordinated  in  the  COEA 
to  ensure  consistency  at  the  milestone. 


Scenarios 

Scenarios  define  how  the  available  intelligence 
baseline  is  applied  in  the  COEA.  DoD  Instruction 
5000.2  requires  that  the  COEA  scenarios  conform  to 
the  DPG  scenarios  discussed  earlier.  The  current  draft 
set  of  IPSs  cover  seven  regional  conflicts.  DIA  assists 
JCS  (J-5)  and  the  Under  Secretary  of  Defense  for 
Policy  in  the  development  of  these  scenarios.  It  is 
important  to  note  that  DIA  does  not  drive  the  scenarios, 
but  recommends  threat  force  levels  and  capabilities. 
Further,  DIA  does  not  validate  the  scenarios;  it 
validates  the  threat  information  in  the  scenarios.  These 
scenarios  are  not  predictions  of  the  fixture,  but  as  a  set, 
help  determine  requirements  for  U.S.  forces  and 
capabilities. 

The  DPG  scenarios  are  not  detailed  enough  to 
conduct  the  COEA  without  additional  assumptions 
regarding  threat  laydowns,  composition  of  threat  units, 
numbers  and  -  -s  of  weapons,  and  weapon 
employment  taci:  and  doctrine.  Such  assumptions 
may  be  drawn  from  the  STAR  and  other  DIA-validated 
sources  or  may  be  developed  on  a  case-by-case  basis. 
Alternative  scenarios  and  threat  excursions  may  be 
considered,  but  differences  with  the  IPSs  must  be 
identified  and  addressed.  DIA  intends  to  develop  a 
Threat  Environment  Projection  document  which 
provides  expanded  threat  environments  for  each  of  the 
major  DPG  scenarios. 

In  the  past.  Service  scenario  developers,  such 
a:  l.v  .Army  Training  and  Doctrine  Command,  have 
taken  the  IPSs  and  expanded  them  to  include  unit 
descriptions,  laydowns,  and  day-by-day  descriptions  of 
the  movement  of  Red  and  Blue  forces.  Vignettes  from 
these  scenarios  have  also  been  used  to  establish  detailed 
test  and  evaluation  (T&E)  criteria.  On  occasion,  DIA 
has  reviewed  and  approved  these  for  use  in  COEAs  and 
T&E.  However,  it  is  not  DIA's  intent  to  validate 
individual  unit  deployments,  but  rather  require  the 
Service  certify  that  deployments  have  been 
accomplished  in  accordance  with  general  military 
principles. 

Threat  Analysis  and  Coordination 

DoD  Manual  5000. 2-M  contains  general 
guidance  for  threat  analysis  in  the  COEA.  These 


guidelines  cover:  consistency  with  the  DPG; 

consideration  of  the  nature,  size,  and  technical 
performance  of  threat  forces;  analysis  in  sufficient 
detail;  implications  of  threat  constraints;  and  the  need 
for  a  range  of  plausible  threats  to  allow  for  uncertainty. 

It  is  important  to  remember  that  there  may  be  widely- 
varying  confidence  levels  associated  with  much  of  the 
available  information. 

Coordination  of  these  and  other  threat  issues 
will  be  improved  by  involving  DIA  early  in  the  COEA 
planning.  DIA  coordination  may  be  handled  differently 
depending  on  the  size  and  nature  of  the  COEA  and  the 
sponsoring  Service,  in  addition  to  local  intelligence 
offices,  threat  coordinating  or  working  groups,  with 
DIA  participation,  may  be  formed  to  review  threat 
baselines  and  identify  additional  threat  assumptions 
required  to  conduct  the  COEA.  Formal  DIA  validation 
will  probably  be  needed.  Early  coordination  will  help 
avoid  delays  and  threat  issues  at  the  milestone. 

Conclusion 

Since  the  promulgation  of  the  DoD  5000  series 
guidance,  DIA  has  become  more  involved  in  the  COEA 
process.  Recent  efforts  include  COEAs  for  the  F-22, 
A-X,  Global  Protection  Against  Limited  Strikes,  and 
Joint  Unmanned  Aerial  Vehicle  programs.  DIA  has 
reviewed  both  digital  threat  models,  as  well  as  the  input 
data  for  these  models,  as  part  of  the  validation  effort. 
Review  of  man-in-the-loop  simulators  which  input  to  a 
COEA  is  a  particularly  new  challenge.  DIA  support 
will  be  ramped-up  as  requirements  are  levied. 
However,  validation  of  threat  models  used  in  the 
COEA,  and  requirements  for  far  more  threat  detail  than 
is  provided  in  the  DPG  and  STAR,  remain  areas  of 
concern.  Validation,  verification,  and  accreditation  of 
models  are  areas  that  the  Services  and  OSD,  with 
support  as  required  from  DIA,  need  to  resolve. 

In  summary,  validation  of  threat  assessments 
for  COEAs  are  part  of  the  threat  support  provided  by 
DIA  for  major  U.S.  weapons  programs.  This  support  is 
based  on  a  requirement  levied  by  DoD  for  DIA 
validation  of  all  threat  information  used  in  such 
programs.  The  COEA  threat  baseline  is  contained  in 
the  DPG  and  special  mission  area  or  system  specific 
threat  documents.  COEAs  should  conform  to  the  set  of 
scenarios  in  the  DPG.  However,  additional  effort  will 
be  required  to  develop  threat  details  for  COEAs.  As 
a  result,  DIA  needs  to  be  involved  early  in  the  COEA 
planning  process  to  avoid  problems  and  delays. 
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2.1.4  Modeling  and  Tools 
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Topic  #3:  Modeling  and  Tools 


Sam  Gardiner 
MORS  Mini-Symposium 

nfifi  |n  tht  Acquisition  ProcM-s  and  th»  Hoi*  ol  OR  In  Pcrtocmino  COEA 

March  9, 1992 


Figure  1:  Changes  in  War  and  Warfare 
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/Requirements  Modeling  and  Tools  |  ^ - ^ 
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[Process  for  Developing  Models  and  Toojsj 
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Figure  2:  Overview 
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Figure  3:  Trends  and  Consequences 


Figure  4:  Lethality 


Figure  5:  Battlefield  Density 


Figure  6:  Flow  of  Information 


Figure  7:  Nonlinear  Continuous  (1) 


Nonllnaar-Cantlnuous 


Figure  8:  Nonlinear  Contiuous  (2) 


Figure  9:  Refusing  Battle 
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Figure  10:  Refusing  Battle 


29 


Figure:  Different  Quality  of  Massing 
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Shock  Individuals  can't  respond 

as  a  unit, 


Disruption 


Dislocation 


Command  is  unable  to  [»DaiJ 
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Response  is  disoriented 
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Destruction 


Unit  is  "dead"  as  a  fighting 
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Figure  13:  Unit  Damage  Concept 


[Ffon^««»rtShl»[0/Slom^jciv7| 


Figure  14:  Campaign  Planning 
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Figure  15:  Campaign,  Phases,  Operations  and  Tasks 
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Figure  16:  Campaign  Measures  of  Effectiveness...? 
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Figure  17:  Non-Continuous  Combat 
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Figure  18:  Analysis  and  Modeling 
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2.1.5  Operational  Effectiveness  Methodologies 


„  MORS  COEA  SYMPOSIUM 

OPERATIONAL 

EFFECTIVENESS 

METHODOLOGIES 

Figure  2:  Operational  Effectivenss  Methodologies 
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Figure  3:  Operational  Effectiveness— A  Historical  Peispective 
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Figure  4:  Operational  Effectiveness— What's  Changed? 
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Figure  5:  The  New  Environment 


•  ID  THE  DECISION  MAKER  AND  HIS  OBJECTIVES 
.  UNDERSTAND  THE  QUESTION 
.  DEFINE  THE  ALTERNATIVES 
.  DEVELOP  THE  METHODOLOGY 
.  QUANTIFY  RESOURCES  NEEDED 
.  GATHER/GENERATE  DATA  (PERF,  COST) 

.  CONDUCT  THE  ANALYSIS 

.  PRESENT  THE  RESULTS  AS  A  COMPLETE  PICTURE 
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Figure  6:  Analytical  Tasks 
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Figure  7:  A  COEA  Study  Mode! 


PROPONENTS  SOMETIMES  OVERLOOK  CRmCAL  PARAMETERS 

OBJECTIVE  ANALYSIS  PRESENTS  A  COMPLETE  PICTURE 
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Figure  9:  Analytical  Tools 


MORS  COEA  SYMPOSIUM 


ANALYTICAL  MODELS 


THETOOLS  ARE  NO  BETTER  THAW  THE  ANALYSTS  USING  THEM 


P.r*mrwnc«  Models  npamlinrul  Models 


Each  Modal  Has  Strengths  and  Weaknessts 
Select  Model  lo  Answer  Questions  •  Don!  Bend 
Questions  to  Fit  the  Mode! 


Figure  10:  Analytical  Models 
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Figure  11:  Analytical  Snakepits 


Figure  12:  The  Study  Process 
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WHAT  DEF^TMENT  QF  DEFENSE  WANTS") 


IN  COMPARING  ALTERNATIVE  SYSTEMS: 

•  ABSOLUTE  VALUES -NOT  RATIOS 

•  DOMINANCE  RELATIONSHIPS 

•  ALTERNATIVES  THAT  ARE  ROUGHLY  EQUAL  IN  COST 

•  LESS  COSTLY  ALTERNATIVES  THAT  ARE  ROUGHLY 
EQUAL  IN  EFFECTIVENESS 


Figure  13:  What  Department  of  Defense  Wants 
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THE  EFFECTIVE  ANALYSIS  \ 


OPERATIONAL  EFFECTIVENESS  -  CONDUCTED  WITHIN  A 
COMBINED  ARMS  FORCE  (JOINT/COMBINED  CONTEXT) 


-  MEASURES  OF  EFFECTIVENESS  fMOEl  ARE  EPR.CE 
RELATED 

-  IF  POSSIBLE,  ESTABLISH  CRITERIA  FOR  MISSION 
SUCCESS -TO  "WIN" 

-  MOE  CAN  BE  QUANTIFIABLE  AND/OR  QUALITATIVE 

-  SYSTEM  MUST  BE  INTEGRATED  INTO  FORCE 

PERFORMANCE  (SYSTEM  EFFECTIVENESS)  -  CONCERNED 
WITH  HOW  WELL  A  SPECIFIC  TYPE  OF  SYSTEM 
PERFORMS  WITHIN  COMBINED  ARMS  FORCE 
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Figure  17:  LH  Operational  Analysis  Methodology 


Figure  18:  LH  MSII  COEA—Typical  Aviation  Unit  Mission  Summaries 
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Figure  19:  LH  Campaign  Analysis-Typical  Operational  Tempo 


Figure  20:  High  Resolution  Combat  Scenario  Library 
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Figure  21:  Operational  Effectiveness  Results 


Figure  22:  LH  MS II  COEA—Histoiy  of  Maintenance  Ratio  Data 
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Figure  23:  Reconnaissance  Squadron 
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Figure  24:  Integration  of  Analytical  Results 
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ANALYTICAL  TRUTH 


A  GOOD  ANALYST  SAID: 

"THE  ROLE  OF  THE  ANAL  YST  IS  TO  REVEAL  THE  TRUTH" 

A  BETTER  ANALYST  SAID: 

"IT  IS  NOT  ENOUGH  TO  REVEAL  TRUTH, 

THE  ANAL  YST  MUST  SELL  IT  AS  TRUTH. " 


B82021M4  Im 


Figure  25:  Analytical  Truth 


46 


2.1.6 


COST  ANALYSIS  IN  COEAs 


Introduction 

Good  morning.  My  name  is  Dan  Nussbaum, 
and  I  am  from  the  Naval  Center  for  Cost  Analysis. 

I  want  to  cover  three  topics  related  to  the  cost 
community  that  I  think  are  important  to  the  COEA 
process: 

-  Identifying  the  role  of  cost  analysis  in 
COEAs,  including  specifying  the  appropriate  cost 
question  and  the  deliverable  from  the  cost  analyst. 

-  Identifying  the  organizations,  by  Service, 
which  are  performing  cost  analysis  to  support  COEA. 

-  Discussing  two  issues  current  in  the  cost 
analysis  community'  that  bear  on  how  future  cost 
estimates  will  be  done:  the  treatment  of  risk  and 
uncertainty  in  cost  estimates  and  the  estimation  of  R&D 
costs. 

A  short  version  of  all  this  is  that  I  want  to 
address  the  issues  of  who  we  are,  what  we  do,  and 
some  questions  that  are  current  within  the  cost  analysis 
community'. 


The  Role  of  Cost  Analysis  in  COEAs 

DoD  5000.2  states  that  there  are  three  reasons 
for  doing  COEAs.  These  reasons  are  to  aid  in  decision 
making,  to  facilitate  communications,  and  to  document 
acquisition  decisions.  My  view  is  that  it  is  the  analyst's 
responsibility  to  give  to  the  decision  maker  a  "guided 
tour"  of  the  decision  space  under  discussion.  That 
means  that  the  analyst  illuminates  choices  for  the 
decision  maker  and  consequences  of  those  decision. 


Whichever  view  one  takes  —  the  DoD  5000.2  view'  or 
the  "guided  tour"  view  —  it  is  clear  that  we  act  in  a 
support  role. 

The  question  often  arises  of  "what  are  the 
proper  costs  to  be  considered  in  a  COEA"?  The  answer 
is  life  cycle  costs  (LCC),  which  are  usually  defined  as 
"the  total  cost  (contractor  and  government)  to  the 
government  for  a  system  over  its  full  life".  While  I 
have  no  intention,  in  this  presentation,  of  doing  a 
tutorial  on  cost  analysis,  the  following  remarks  are 
pertinent  and  included  for  completeness: 

-  LCC  covers  all  phases  of  the  life  cycle: 
development,  procurement,  operation  and  support,  and 
disposal 

-  Each  phase  of  the  life  cycle  has  its  own  work 
breakdown  structure  (WBS).  Further,  under  each  WBS 
element  in  each  life  cycle  phase,  there  is  a  further 
refinement  of  the  WBS. 

-  The  usual  WBS  (as  structured  in  MIL  STD 
881)  is  used  for  estimating  costs  of  either  weapon 
systems  or  automated  information  systems.  There  are 
other  settings  for  cost  estimating,  including  force 
structure  issues,  Defense  Management  Review 
initiatives,  and  personnel  and  logistics  structure  issues 
(e.g.,  umt-versus  individual  personnel  rotation  policies). 

There  is  a  standard  picture  of  the  phases  of 
LCC  reproduced  below.  It  serves  the  purpose  of 
showing  the  timing  of  life  cycle  phases,  the  relative 
magnitude  of  the  costs  incurred  in  each  life  cycle  phase, 
and  it  provides  some  nomenclature.  For  these  purposes, 
the  chart  is  fine.  There  are,  however,  some  aspects  that 
are  misleading: 


-  The  peak  of  the  O&S  curve  is  higher  than  the 
peak  of  the  Production  cost  curve.  In  fact,  while  total 
O&S  costs  are  usually  greater  than  total  Production 
costs,  the  highest  cost  year  for  O&S  costs  is  usually 
less  than  the  highest  cost  year  for  Production  costs,. 

-  For  software  projects,  the  correct  relative 
magnitude  for  costs  is:  Development  costs  >  O&S  costs 
>  Production  costs 

-  Note  also  that  the  new  nomenclature  for  the 
Full  Scale  Development  (FSD)  phase  is  now 
Engineering  Management  and  Development  (EM&D). 

What  are  the  products  that  you  can  expect 
from  the  cost  analysis  community  —  the  deliverables  by 
cost  analysts  —  in  a  COEA?  These  deliverables  include: 

-  LCC  by  appropriation  for  each  alternative, 
done  in  constant  $  and  then  year  $ 

-  Present  value  analysis  for  each  alternative. 
While  tr  -e  are  different  viewpoints  on  the  utility  of 
present  .e  analysis,  it  is  my  opinion  that  it  is  a  useful 
analytic*.,  tool.  Its  value  lies  in  the  fact  that  it  provides 
a  single-valued  discriminant  across  all  the  options  that 
are  under  consideration. 

-  Risk  and  uncertainty  analyses.  These 
analyses  represent  excursions  on  relevant  issues  and 
provide  to  the  decision  maker  a  measure  of  the 
robustness  of  the  preferred  solution. 

-  Documentation.  Not  only  does  5000  require 
it,  but  we  want  to  document  our  analysis  as  part  of  the 
general  documentation  of  the  decisions.  The  essence  of 
science  is  reproducibility,  and  it  is  that  reproducibility 
of  the  COEA  results,  assembled  and  composed  in  the 
documentation,  that  permits  later  analysts  to  recreate 
our  results.  Documentation  should  be  written  to  such 
a  level  that  a  journeyman  analyst  can  reproduce  the 
estimate. 

I  want  to  emphasize  the  point  that  cost 
analysis  has  to  be  involved  early  in  the  COEA  process. 
Here  is  a  flow  of  the  cost  analysis  process. 


I  want  to  emphasize  "factors  which  impact", 
because  it  makes  an  important  point  about  how  the  cost 
analysis  community  operates.  Among  the  factors  winch 
influence  the  cost  estimate  are: 

-  mission,  technology  required,  special  tooling 
and  test  equipment  required,  configurations  for  the 
system  and  its  support,  acquisition  concept  (sole  source, 
buy-out,  leader-follower),  physical  and  performance 
characteristics,  quantities,  production  rates,  and 
logistics  concept. 

The  point  is  that  all  those  issues  that  are  of 
interest  to  engineers  at  the  early  stages  of  a  project  are 
also  of  interest  to  the  cost  analyst.  Three  observations 
follow  from  this 

-  There  must  be  a  baseline  for  the  program, 
which  identifies  the  details  and  specifications  of  the 
program. 

-  The  baseline  document  is  known  as  a  Cost 
Analysis  Requirements  Document  (CARD)  and  is 
called  for  in  the  new  DoDI  5000.4.  A  CARD  should  be 
prepared  for  each  alternative  under  consideration  in  the 
COEA. 

-  Cost  analysis  must  be  brought  into  the 
program  at  the  earliest  possible  stage. 
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Organizational  Location  of  Cost  Analysis  Capability 

Only  10%  of  the  attendees  at  this  conference 
are  in  the  formal  cost  analysis  community,  so  for  the 
sake  of  the  other  90%,  I  want  to  identify  the  location  of 
the  Services'  cost  analysis  capability.  The  truth  is  that 
the  next  three  charts  —  which  purport  to  show  the 
sendees'  cost  estimating  organizations  and  where  they 
are  located  in  the  bureaucracy  —  are  only  roughly  right. 
I  am  sure  they  are  "precisely  wrong".  Still,  they  are 
approximately  correct,  and  I  think  one  of  the  goals  of 
this  conference  is  to  get  a  good  handle  on  these 
structures. 


COST  ANALYSIS  SUPPORT  TO  ACAT  I 

TYPICAL  USAF  COLA  STRUCTURE 


The  cost  analysis  capability  within  the  Air 
Force  resides  in  the  Major  Commands  and  within  the 
Secretariat  (SAF/FM).  Within  the  MAJCOMs,  the  FM 
shops,  the  Air  Logistics  Centers  and  the  cost 


the  Systems  Commands  cost  shops  (AIR-524  and  SEA- 
017),  some  of  the  Navy  Labs,  and  the  Center  for  Naval 
Analysis  (CNA).  The  observations  to  make  about  the 
organization  of  the  Navy's  cost  analysis  capabilities  are 

-  The  capabilities  are  split  between  the 
Secretariat  and  the  Blue  Suit  side  of  the  house. 

-  NCA  does  milestone  independent  cost 
estimates,  and  by  sitting  on  the  COEA  oversight  board, 
provides  for  a  consistency  check  on  cost  estimates. 


The  cost  analysis  capability  within  the  Army 
resides  m  the  Army  Cost  and  Economic  Analysis 
Center  (US  ACE  AC),  the  Army  Materiel  Command 
(AMC)  and  AMC's  Major  Subordinate  Commands 
(akin  to  the  Navy's  System  Commands).  The 
observations  to  make  about  the  organization  of  the 
Army's  cost  analysis  capabilities  is  that  there  is  a 
dispersed  cost  analysis,  and  that  the  capabilities  are 
split  between  the  Secretariat  and  the  Green  Suit  side  of 
the  house. 


Current  Issues  in  Cost  Analysis 

Now  I  want  to  turn  my  attention  to  some  of 
the  issues  that  are  of  high  interest  in  the  cost  analysis 
community  right  now.  I  start  with  some  slides  about 
cost  growth  and  schedule  growth.  My  thanks  to  Mr. 

Dave  Olsen  of  The  Analytical  Sciences  Corporation  for 
these  slides.  As  you  look  at  these  slides,  the  salient 
points  to  recognize  are  that: 

-  Cost  growth  exists  for  private  sector  as  well 
as  public  sector  projects 

-  Schedule  growth  exists  for  private  sector  as 
well  as  public  sector  projects 

-  Weapons  system  cost  and  schedule  growth 
compares  favorably  to  cost  and  schedule  growth 
experienced  by  non-weapon  system  projects,  contrary1 
to  popular  mythology. 

The  third  chart,  which  provides  descriptive 
statistics  of  the  duration  of  life  cycle  phases  for 
different  classes  of  weapon  systems,  represents  the 
beginning  of  research.  In.  fact,  from  these  data,  we  can 
model  weapon  system  ph.  duration,  and  thereby  have 
a  basis  from  which  to  do  risk  analysis. 

Now  I  want  to  turn  to  some  basic  definitions 
in  risk  and  uncertainty  .  While  these  definitions  are  not 
universally  held  to,  they  do  have  some  currency  within 
the  cost  analysis  community. 

-  "Risk"  is  about  things  that  happen  to  the 
program,  such  as  changes  in  program  assumptions, 
performance,  schedules,  weight,  acquisition  strategy,  or 
vendor  business  base.  Usually,  the  impact  of  risk  upon 
the  cost  estimate  is  handled  by  sensitivity  analysis. 

-  "Uncertainty"  is  the  embodiment  of  the 
statistical  nature  of  cost  estimating  relationships 
(CERs).  We  can  think  of  uncertainty  as  the  "fog  on  my 
(cost  estimator's)  glasses"  due  to  the  statistical  nature  of 
CERs. 

Everybody  is  interested  in  quality,  including 
cost  analysts  who  care  about  quality  in  their  cost 
estimates.  I  think  of  uncertainty  as  the  quality 
jsurance  of  cost  analysis. 


One  of  the  issues  that  has  always  raged  within 
the  cost  analysis  community  is  the  question  of  whether 
we  better  serve  decision  makers  by  producing  point 
estimates  or  range  estimates.  Point  estimates  are 
attractive  in  that  they  conform  to  the  input  formats  for 

both  the  DoD  budget  and  congressional  processes,  and 

they  are  easy  to  deal  with.  Range  estimates  have  the 
advantage  that  they  conform  to  the  uncertainty  of  the 
world,  and  they  provide  an  explicit  recognition  of  the 
statistical  underpinnings  of  CERs. 

Here  is  an  example  of  the  use  of  risk  analysis 
in  cost  analysis.  The  estimation  problem  was  to 
estimate  the  cost  of  an  unmanned  air  vehicle  (UAV). 
The  base  case  for  the  analysis  was  a  weight  of  1900 
pounds,  and  a  life  expectancy  of  25  flights  per 
MRUAV.  There  was  technical  risk  that  the  true  weight 
of  the  UAV  —  once  the  vehicle  was  actually  built  — 
could  be  as  low  as  1750  pounds  or  as  high  as  2100 
pounds.  There  was  also  a  technical  risk  that  the  life 
expectancy  would  more  closely  reflect  the  lower, 
historical  experience  in  other,  similar  programs. 
Therefore,  sensitivity  excursions  where  run  at  life 
expectancies  of  1 2,  20,  and  25  flights.  The  following 
CER,  relating  the  cost  of  the  100th  UAV  to  its  weight, 
was  developed: 

Cost  of  unit  100(FY89$K)  =  233.6  + 

.20 1  *(max  weight) 

The  following  figures  display  the  sensitivity  of  the  cost 
estimate  to  these  risk  factors. 
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The  lessons  to  be  taken  from  Figures  5  and  6 
are  that  the  cost  estimate  is  not  sensitive  too  the  weight 
variable,  but  that  it  is  very  sensitive  to  the  attrition 
variable. 


Another  issue  of  importance  in  working  to 
improve  the  quality  of  cost  estimates  is  to  measure  the 
impact  on  a  cost  estimate  of  a  change  in  production 
rate,  especially  drastic  changes.  The  obvious  problem 
is  that  the  fixed  cost  at  the  manufacturing  facility  have 
to  be  spread  over  fewer  manufactured  units.  We  have 
a  study  in  progress  now  to  examine  this  problem  in  the 
case  of  the  F/A-18E/F. 


There  is  much  to  do  in  the  area  of  cost 
estimating  risk  and  uncertainty  analysis.  Here  is  a  short 
list  of  risk  and  uncertainty’  operations  research 
questions: 

•  identify’  appropriate  distributions  and  models  for 

-  R&D  schedule  slippage 

-  weight  growth 


-  business  base  implications 
•  determine  correlations  among  cost  variables 

Now  I  want  to  transition  to  the  new  topic  of 
current  issues  in  R&D  estimating. 

As  DoD  budgets  in  general  and  Procurement 
budgets  in  particular  are  downsized,  there  may  be  an 
increased  emphasis  on  R&D  and  concomitant 
deemphasis  on  procurement.  From  a  cost  estimator's 
perspective,  the  interesting  question  is  what  impact 
such  an  adjustment  to  procurement  policy  will  have  on 
current  cost  estimating  models.  Since  current  cost 
estimating  models  are  built;t  upon  historical  data,  such 
a  policy  change  would  seem  to  undercut  the  validity 
these  models. 

A  second  important  issue  is  that  of  measuring 
the  impacts  of  having  programs  being  done  for  more 
than  one  Service  (Jointness).  Many  programs  - 
hardware  as  well  as  organizational  support  structures 
(e.g.,  finance  and  accounting)  —  are  now  operating  in 
a  joint  arena.  The  cost  unpact  of  this  "purpling"  of 
programs  and  services  is  yet  to  be  characterized. 

Here  are  some  operations  research  questions 
in  the  area  of  R&D  cost  estimating: 

-  Should  we  over-weight  R&D  costs  in 

COEA? 


-  What  new  techniques  are  needed  to  estimate 
R&D  costs? 

-  What  is  the  proper  weighting  of  R&D  costs 
m  a  COEA? 


What  are  the  proper  time -phased 
distributions  for  R&D  costs? 

-  What  comparisons  can  be  drawn  between  the 
Development  and  Validation  cost  for  a  program 
developed  by  a  single  service  compared  to  one 
developed  in  a  joint  environment? 

-  What  are  the  costs  to  resurrect  a  shelved 
R&D  program  and  to  bring  it  into  production  status? 

Finally,  I  want  to  make  a  plea  for  keeping 
open  the  cost  data  pipeline.  Cost  data  from  programs  is 
the  sine  qua  non  of  cost  estimating,  in  the  sense  that 
CERs  and  cost  factors  are  grounded  in  historical 


experience  (the  past  is  prologue).  In  fact,  data  are 
needed  at  the  WBS  level,  so  that  estimation  may  also  be 
done  there.  These  data  are  available  from  cost  reports 
(CCDRs,  CPRs  and  so  forth).  Cost  analysts  often  hear 
the  argur  ent  that  cost  reports  are  useless  from  FFP 
contracts  since  the  contractor  bears  all  the  risk.  The 

argument  is  specious  (the  A- 12  and  P-7  contracts  dispel 
the  myth),  and  in  fact  harmful  to  good  program 
management. 

Summary 

A  summary  of  what  I  have  said  in  this 
presentation  is: 

-  Involve  cost  analysts  early  in  the  project. 
Among  the  factors  which  influence  the  cost  estimate  are 
mission,  technology  required,  special  tooling  and  test 
equipment  required,  configurations  for  the  system  and 
its  support,  acquisition  concept  (sole  source,  buy-out, 
leader-follower),  physical  and  performance 
charact-  sties,  quantities,  production  rates,  and 
logistic  concept.  The  point  is  that  all  those  issues  that 
are  of  mterest  to  engineers  at  the  early  stages  of  a 
project  are  also  of  mterest  to  the  cost  analyst. 

-  Force  a  CARD  for  ACAT  I  grams.  The 
baseline  document  is  known  as  a  Cost  Analysis 
Requirements  Document  (CARD)  and  is  called  for  in 
the  new  DoDI  5000.4.  A  CARD  should  be  prepared  for 
each  alternative  under  consideration  in  the  COEA. 


There  must  be  a  baseline  for  the  program,  which 
identifies  the  details  and  specifications  of  the  program. 

-  Uncertainty  is  the  QA  of  our  business. 
Uncertainty  is  the  embodiment  of  the  statistical  nature 
of  cost  estimating  relationships  (CERs).  We  can  think 

of  uncertainty  as  the  "fog  on  my  (cost  estimator's) 
glasses"  due  to  the  statistical  nature  of  CERs. 
Everybody  is  interested  m  quality,  including  cost 
analysts  who  care  about  quality  in  their  cost  estimates. 

-  Risk  analysis  provides  texture  and  context  to 
the  results.  Risk  is  about  things  that  happen  to  the 
program,  such  as  changes  in  program  assumptions, 
perfoimance,  schedules,  weight,  acquisition  strategy,  or 
vendor  business  base.  Usually,  the  impact  of  risk  upon 
the  cost  estimate  is  handled  by  sensitivity  analysis. 

-  R&D  costs  to  gain  in  importance.  As  DoD 
budgets  in  general  and  Procurement  budget?  m 
particular  are  downsized,  there  may  be  an  increased 
emphasis  on  R&D  and  concomitant  deemphasis  on 
procurement.  From  a  cost  estimator's  perspective,  the 
interesting  question  is  what  impact  such  an  adjustment 
to  procurement  policy  will  have  on  current  cost 
estimating  models.  Since  current  cost  estimating 
models  are  built  upon  historical  data,  such  a  policy 
change  would  seem  to  undercut  the  ilidity  these 
models. 


2.1.7  INTEGRATING  COST  AND 
EFFECTIVENESS  IN  COST  AND  OPERATIONAL 
EFFECTIVENESS  ANALYSES 

INTRODUCTION.  The  analysts  have 
estimated  both  effectiveness  and  cost  for  each 
alternative  considered  in  the  Cost  and  Operational 
Effectiveness  Analysis  (COEA).  Their  next  task  is  to 
rank  order  the  alternatives.  Ranking  the  alternatives 
requires  that  the  distinct  attributes,  cost  and 
effectiveness,  somehow  be  integrated.  This  paper 
considers  the  integration  problem  from  the  viewpoint 
of  multiattribute  or  multiple  criteria  decision  making. 
From  this  viewpoint  an  individual  decision  maker's 
preferences  and  value  trade-offs  among  multiple  and 
often  conflicting  attributes  can  be  explicitly  modeled. 
This  model  can  then  be  used  to  rank  the  decision 
alternatives.  From  the  literature  several  possible 
techniques  are  identified.  Two  of  the  techniques  are 
demonstrated  in  the  context  of  an  abbreviated  COEA 
on  the  TOW  Sight  Improvement  Program. 

Multiattribute  Decision  Making.  The 

literature  provides  a  wealth  of  possible  multiattribute 
decision  making  techniques.  Consideration  is  given  to 
prescriptive  techniques  in  contrast  to  descriptive 
techniques.  Prescriptive  techniques,  largely  the 
concern  of  operations  research,  seek  to  help  people 
make  better  decisions.  Such  techniques  are  founded 
on  compelling  assumptions  or  practical  decision 
criteria.  On  the  other  hand,  descriptive  techniques 
seek  to  describe  decision  making  behavior.  Primary 
references  are  Arrow  and  Raynaud,  Chankong  and 
Haimes,  Hwang  and  Yoon,  Keeney  and  Raiffa, 
MacCrimmon,  and  Saaty. 

In  a  recent  study  Cassady  and  Goodwin 
reviewed  multiattribute  decision  making  techniques 
for  possible  use  in  COEA.  Table  1  lists  the 
techniques  that  they  reviewed. 

Based  on  their  theoretical  soundness,  ease  of 
use,  data  requirements,  and  prevalence,  five  of  the 
techniques  listed  in  Table  1  were  found  to  be 
appropriate  for  general  use  in  COEA.  These  five  are 
listed  in  Table  2. 


Table  1 

Multiattrribute  Decision  Techniques  Reviewed 

Dominance 

Maximin 

Maximax 

Majority  Rule 

Holer's  Ranking  Technique 

Conjunctive  Technique 

Disjunctive  Technique 

Stochastic  Dominance 

Lexicographic 

Lexicographic  with  Minima 

Key  Attribute 

ELECTEE 

Permutation 

Analytic  Hierarchy  Process  (AHP) 
Multiattribute  Value  Theory  (MAVT) 
Simple  Additive  Weighting  (SAW) 
Technique  for  Order  Preference  by  Similarity 
To  Ideal  Solution  (TOPSIS) 
Cost-effectiveness  ratios 
Multiattribute  Utility  Theory  (MAUT) 

Table  2 

Multiattribute  Decision  Techniques 
Appropriate  for  General  Use  in  COEA 

Dominance 

Conjunctive  Technique 
Analytic  Hierarchy  Process  (AHP) 
Multiattribute  Value  Theory  (MLAVT) 
Multiattribute  Utility  Theory  (MAUT) 

The  fact  that  a  technique  was  not  found 
appropriate  for  general  use  does  not  mean  that  it 
should  never  be  considered  for  use.  In  particular 
simple  ranking  techniques  such  as  majority  rule  and 
Koler's  ranking  technique  have  great  potential. 
However,  further  research  is  required  for  a  full 
understanding  of  their  application  in  a  COEA 
environment.  On  the  other  hand,  techniques  such  as 
cost-effectiveness  ratios  or  TOPSIS,  were  found  not 
appropriate  for  general  use  in  COEA  because  of  their 
unique  underlying  assumptions.  If  these  assumptions 
hold  for  a  particular  application  then  their  use  should 
be  considered. 
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TSIP  COEA.  In  their  study  Cassady  and 
Goodwin  demonstrated  AHP  and  MAUT  in 
conjunction  with  the  recent  TOW  Sight  Improvement 
Program  (TSIP)  Abbreviated  Cost  and  Operational 
Effectiveness  Analysis  (COEA).  This  COEA  was 
conducted  to  scrutinize  alternative  antiarmor 
capabilities  in  mechanized  and  light  infantry. 

The  demonstration  considered  only  a  subset 
of  alternatives  from  the  mechanized  infantry  portion 
of  the  COEA.  Mechanized  infantry  battalions  include 
four  mechanized  infantry  companies  fielding  Bradley 
Fighting  Vehicles  (BFV)  firing  TOW  missile.  In 
addition,  the  battalion  contains  an  antiarmor  company 
with  improved  TOW  vehicles  (ITV)  also  firing  TOW. 
The  TOW  sight  improvement  program  sought  to 
replace  the  current  TOW  sight  with  a  new  sight. 
Tabis  3  lists  the  alternatives  considered. 


Table  3 
Alternatives 
Mech  Inv  Co 
BFV  TOW  Sight 
Alternative 

Base  Case  Current 

Alt  1  Current 

Alt  2  New 

Alt  3  New 


Anti  Amor  Co 

rrv  tow  sight 

Current 

New 

Current 

New 


Alternatives  were  compared  by  cost  and 
effectiveness.  Cost  was  assessed  using  twenty  year 
life  cycle  cost  in  constant  FY92  dollars.  Effectiveness 
was  assessed  using  loss  exchange  ratios  (LER)  from 
a  high  resolution  combat  simulation  model.  Three 
distinct  combat  scenarios  were  represented:  European 
brigade  meeting  engagement  (EUR  MGT),  European 
balanced  task  force  defense  (EUR  DEF),  and 
Southwest  Asian  brigade  meeting  engagement  (SWA 
MGT).  Figure  1  illustrates  the  COEA  attributes  in  a 
hierarchical  fashion. 

Analytic  Hierarchy  Process.  AHP  proceeds 
with  the  decision  maker  scaling  attributes  and 
alternatives.  Scale  values  are  intended  to  capture  the 
relative  importance  of  an  attribute  or  alternative  in 
the  hierarchy.  Scale  values  are  converted  to 
comparable  weights  using  an  eigenvector  method. 
The  weights  are  then  combined  linearly  up  the 
hierarchy.  Several  commercial  software  packages  are 
available  for  the  application  of  AHP.  The  scale  used 
to  assess  the  pairwise  importance  of  attributes  or 
alternatives  is  illustrated  in  table  4. 


PROVIDE  AN  INFANTRY 
ANTIARMOR  CAPABILITY 


COST 


EFFECTIVE  ANTIARMOR 
CAPABILITY 


BC  ALT  ALT  ALT  EUR  MGT 
123  LER 


EUR  DEF 
LER 


SWA  MGT 
LER 


BC  ALT  ALT 
1  2 


ALT  BC  ALT  ALT  ALT  BC  ALT  ALT 

3  12  3  12 


ALT 

3 


Figure  1.  The  Attribute  Hierarchy 
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Table  4 
AHP  Scale  of 
Pairwise  Importance 


Importance  of  one 

factor  over  another  Value 

Equal  1 

2 

Weak  3 

4 

Strong  5 

6 

Very  Strong  7 

8 

Absolute  9 


The  AHP  importance  scale  also  makes  use  of 
the  reciprocals  of  the  values  in  Table  4.  For  example, 
if  the  pairwise  importance  of  factor  1  over  factor  2  is 
judged  to  be  weak  and  valued  at  3,  the  pairwise 
importance  of  factor  2  over  factor  1  is  valued  at  1/3. 
The  TSIP  COEA  hierarchy  requires  six  distinct 
scalings.  At  the  top  the  decision  maker  provides  scale 
values  on  the  relative  importance  of  cost  and 
effectiveness  in  meeting  the  overall  decision  criterion. 
Continuing  down  the  hierarchy,  the  decision  maker 
scales  the  relative  importance  of  the  three  LER 
attributes  in  terms  of  their  contribution  to  the 
effectiveness  attribute.  Next  the  alternatives  are  scaled 
in  terms  of  their  contribution  to  the  cost  attribute,  the 
European  meeting  LER  attribute,  the  European 
Defense  LER  attribute,  and  the  SWA  meeting  LER. 

For  demonstration  purposes  hypothetical 
scale  values  are  used.  While  it  is  hoped  that  these  are 
reasonable  values,  no  claim  is  made  to  their  empirical 
validity.  Table  5  gives  the  scale  values  and 
eigenvector  weights  for  the  relative  importance  of  the 
contribution  of  cost  and  antiarmor  capability  to  the 
decision.  This  choice  of  scale  values  weights  cost  and 
effectiveness  equally. 


Table  5 

Pairwise  Comparisons  of 
Cost  and  Antiaimor  Capability 


Capability 

Cost 

Anti  Aimor 

Cost 

1 

1 

Antiaimor  Capability 

1 

1 

Eigenvector  weights  -  (0.50,  0.50) 

Table  6  gives  scale  values,  principal 
eigenvalue,  and  eigenvector  weights  for  the  relative 
importance  of  the  three  LER  attributes  in  their 
contribution  to  the  antiarmor  capability.  The  scale 
values  have  been  chosen  so  that  the  first  two  LER 
attributes  are  equally  valued  and  the  third  is  weakly 
more  important. 

Table  6 

Pairwise  Comparisons  of 
LER  Attributes 


EURMGT 

EUR  DEF 

SWA  MGT 

LER 

LER 

LER 

EUR  MGT  LER  1 

1 

1/3 

EURDEFLER  1 

1 

1 

SWA  MGT  LER  1 

1 

1 

Eigenvalue  -  3.00,  Weights  -  (0.20,  0.20,  0.60) 

The  principal  eigenvalue  is  a  measure  of  the 
consistency  of  the  scale  values.  Its  value  is  at  least  as 
large  as  the  number  of  factors  scaled.  Complete 
consistency  occurs  when  this  minimum  value  is 
obtained.  In  the  present  case  the  value  of  3.00 
indicates  that  the  scale  values  are  completely 
consistent 
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Table  7  gives  scale  values,  principal 
eigenvalue,  and  eigenvector  weights  for  the  cost  of 
the  alternatives.  These  scale  values  are  derived  from 
cost  estimates  (inFY92  constant  million  dollars).  The 
scale  values  have  been  chosen  so  that  the  base  case 
is  valued  strongly  over  alternatives  one  and  two, 

alternatives  one  and  two  are  equally  valued,  and 
alternatives  one  and  two  are  valued  strongly  over 
alternative  three.  In  addition  the  base  case  is  valued 
absolutely  over  alternative  three. 

Table  7 

Pairwise  Comparisons  of 
Cost  of  Alternatives 


Base 

Alt 

Alt 

Alt 

Case 

1 

2 

3 

Cost 

350 

900 

982 

1344 

Base  Case 

1 

5 

5 

9 

Alt  1 

1/5 

1 

1 

5 

Alt  2 

1/5 

1 

1 

5 

Alt  3 

1/9 

1/5 

1/5 

1 

Table  8 

Pairwise  Comparisons  of 
European  Meeting  LER  Estimates 


Base 

Alt 

Alt 

Alt 

Case 

1 

2 

3 

Median  LER 

1.00 

1.04 

1.40 

1.77 

Base  Case 

l 

1 

1/5 

1/9 

Alt  1 

l 

1 

1/5 

1/9 

Alt  2 

5 

5 

1 

1/5 

Alt  3 

9 

9 

5 

1 

Eigenvalue  -  4.13,  Weights  -  (0.06,  0.06,  0.22,  0.66) 

A  principal  eigenvalue  of  4. 13  indicates  that 
the  scale  values  are  highly  consistent. 

In  Table  9  alternatives  one  and  two  are 
equally  valued.  Alternatives  one  and  two  are  valued 
weakly  over  the  base  case  and  alternative  three  is 
value  weakly  over  alternatives  one  and  two.  Also 
alternative  three  is  valued  strongly  over  the  base  case. 


Eigenvalue  -  4.13,  Weights  -  (0.64,  0.16,  0.16,  0.04) 

A  principal  eigenvalue  of  4.13  indicates  that 
the  scale  values  are  highly  consistent. 

Tables  8,  9,  and  10  give  scale  values,  the 
principal  eigenvalue,  and  the  eigenvector  weights  of 
the  alternatives  for  the  LER  attributes.  These  scale 
values  are  based  on  median  LER  estimates.  In  Table 
8  the  base  case  and  alternative  one  are  valued  the 
same.  Alternative  two  is  valued  strongly  over  the 
base  case  and  alternative  one.  Alternative  three  is 
valued  strongly  over  alternative  two.  Also  alternative 
three  is  valued  absolutely  over  the  base  case  and 
alternative  one. 


Table  9 

Pairwise  Comparisons  of 
European  Defense  LER  Estimates 


Base 

Alt 

Alt 

Alt 

Case 

1 

1 

1 

Median  LER 

2.29 

2.44 

2.38 

2.57 

Base  Case 

1 

1/3 

1/3 

1/5 

Alt  1 

3 

1 

1 

1/3 

Alt  2 

3 

1 

1 

1/3 

Alt  3 

5 

3 

3 

1 

Eigenvalue  -  4.04,  Weights  -  (8.08,  0.20,  0.20,  0.52) 

A  principal  eigenvalue  of  4.04  indicates  that 
the  scale  values  are  highly  consistent. 
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In  Table  10  the  base  case  and  alternative  one 
are  equally  valued.  Alternative  two  is  valued  weakly 
over  the  base  case  and  alternative  one.  Alternative 
three  is  valued  weakly  over  alternative  two  and 
strongly  over  the  base  case  and  alternative  one. 

Table  10 

Pairwise  Comparisons  of 
SWA  Meeting  LER  Estimates 


Base 

Alt 

Alt 

Alt 

Case 

1 

2 

3 

Median  LER 

1.34 

1.37 

1.49 

1.62 

Base  Case 

1 

1 

1/3 

1/5 

Alt  1 

1 

1 

1/3 

1/5 

Alt  2 

3 

3 

1 

1/3 

Alt  3 

5 

5 

3 

1 

Eigenvalue  -  4.04,  Weights  -  (0.10,  0.10,  0.24,  0.56) 


and  effectiveness  equally.  Had  effectiveness  been 
valued  just  very  weakly  over  cost  the  scale  values  of 
Table  5  would  be  replaced  by  those  of  Table  12 
below. 


Table  12 

Revised  Pairwise  Comparisons  of 
Cost  and  Antiaimor  Capability 


Cost 

Antiarmo  r 

Capability 

Cost 

1 

1/2 

Antiaimor  Capability 

2 

1 

Eigenvector  Weights  -  (1/3,  2/3) 

With  these  new  weights  for  cost  and 
effectiveness  the  overall  weights  of  the  alternatives 
become  those  of  Table  13. 


A  principal  eigenvalue  of  4.04  indicates  that 
the  scale  values  are  highly  consistent. 

The  final  step  in  the  AHP  is  to  aggregate  the 
separate  weights  into  an  overall  weight  for  each 
alternative.  The  following  linear  format  is  used. 

AHP  Weight  =  0.5  *  (Cost  Weight) 

+  0.5  *  0.2  *  (EUR  MGT  LER  Weight) 
+  0.5  *  0.2  *  (EUR  MGT  LER  Weight) 
+  0.5  *  0.6  *  (SWA  MGT  LER  Weight) 

Table  1 1  gives  the  overall  AHP  weight  for 
each  alternative. 


Table  13 

Revised  AHP  Weights  of  Alternatives 


Alternative 

Weight 

Base  Case 

0.27 

Alt  1 

0.12 

Alt  2 

0.21 

Alt  3 

0.40 

In  this  case 

alternative  three  r: 

:  first 

followed  by  the  base  case.  A  simple  computation 
shows  that  if  effectiveness  is  valued  at  1.21  over  cost 
the  base  case  and  alternative  three  tie  for  the  first 
rank  with  equal  weights. 


Table  11 

AHP  Weights  of  Alternatives 


Alternative 

Weight 

Base  Case 

0.36 

Alt  1 

0.13 

Alt  2 

0.20 

Alt  3 

0.31 

The  base  case 

ranks  first  followed  by 

alternative  three.  The  greater  effectiveness  of 
alternative  three  is  offset  by  its  greater  cost.  This 
results  in  part  from  valuing  the  importance  of  cost 


MultiattributeUtility  Theory. The  application 
of  MAUT  proceeds  in  three  steps:  modeling  the 
decision  maker's  utility  over  the  attributes,  developing 
probability  distributions  for  outcomes  of  alternatives, 
and  integrating  utility  with  the  distributions.  Several 
commercial  software  packages  are  available  for  the 
application  of  MAUT. 

To  simplify  the  modeling  of  utility  for  the 
demonstration  it  is  assumed  that  the  multiattribute 
utility  function  decomposes  as  a  sum  of  products  of 
single  attribute  utility  functions.  It  is  also  assumed 
that  the  decision  maker  is  risk  neutral  with  regard  to 
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Table  14 

Single  Attribute  Utilities 


Attribute 

Range 

Utility 

Scaling 

Constant 

Cost 

200  to  1500 

UC(c)  =  (1500-c)/1300 

KC 

EUR  MGT  LER  0.25  to  4.0 

UM(x)  =  4(x-0.25)/15 

KM 

EUR  DEF  LER 

0.25  to  4.0 

UD(y)  =  4(y-0.25)/15 

KD 

SWA  MGT  LER 

0.25  to  4.0 

US(z)  =  4(z-0.25)/15 

KS 

Using  the  four  scaling  constants  introduced  in  Table  14  die  utility  function,  U(c,x,y,z),  decomposes  as 


U(C,x,y,z)  =  KC*UC(c)  +  KM*UM(x)  +  KD*UD(y)  +  KS*US(z) 

+  K*[KC*KM*UC(c)*UM(x)  +  KC*KD*UC(c)*UD(y)  + 
KC*KS*UC(c)*US(z)  +  KM*KD*UM(x)*UD(y)  + 
KM*  KS  *  UM(x)  *  US  (z)  +  KD*KS*UD(y)*US(z>] 

+  K*K*[KC*KM*KD*UC(c)*UM(x)*UD(y)  + 
KC*KM*KS*UC(c)*UM(x)*US(z)  + 
KC*KD*KS*UM(c)*UD(y)*US(z)  + 

KM*KD  *KS*UM(x)  *UD(y)  *US(z)] 

+  K*K*K*  [KC*KM*KD  *KS  *UC(c)  *UM(x)  *UD(y)  *US(z)] 


each  single  attribute.  As  a  consequence  the  single 
attribute  utility  functions  are  linear.  Formal  checking 
of  these  assumptions  would  be  required  in 
practice.  Their  failure  to  hold  would  require 
more  complex  techniques.  Table  14  gives  the  single 
attribute  utility  functions  which  are  scaled  from  0  to 
1  over  their  range. 

Next  the  four  scaling  constants,  KC  through 
KS,  are  determined.  This  requires  detailed  and 
complex  assessment  sessions  with  the  decision  maker. 
Software  such  as  IDEA  can  be  useful  in  simplifying 
this  assessment.  After  the  four  scaling  constants,  KC 
through  KS,  are  determine,  the  constant  K  is 
determined  numerically  as  the  solution  of  a 
polynomial.  Table  15  gives  hypothetical  values  for 
the  scaling  constants.  The  scaling  constants,  KC 
through  KS,  have  a  utility  interpretation  at  the 
extreme  values  of  the  attributes.  They  are  not  to  be 
interpreted  as  the  relative  importance  of  an  attribute, 
i.e.,  KC  is  not  the  relative  importance  of  the  cost 
attribute. 


Table  15 

MAUT  Scaling  Constants 


Constant 

Utility  Equivalent 

Value 

KC 

U(200, 0.25, 0.25, 0.25) 

0.200 

KM 

U(1500, 4.0, 0.25, 0.25) 

0.300 

KD 

U(1500, 0.25, 4.0,0.25) 

0.300 

KS 

U(1500, 0.25, 0.25, 4.0) 

0.400 

K 

None 

-0.409 

The  next  step  is  to  develop  probability 
distributions  for  outcomes  of  alternatives.  It  is 
assumed  that  the  attributes  for  any  particular 
alternative  are  independent.  Formal  checking  of  this 
assumption  would  be  required  in  practice.  Its  failure 
to  hold  would  require  more  complex  techniques. 
Because  of  the  assumptions  of  independence  and  of 
risk  neutrality  only  the  means  of  the  distributions  of 
outcomes  are  required.  Table  16  gives  the  necessary 
values. 


Table  16 

Means  of  Outcomes 


Attributes 


EURMGT 

EUR  DEF 

SWAMGT 

COST 

LER 

LER 

LER 

Alternative 

Base  Case 

350 

1.06 

2.35 

1.30 

Alt  1 

900 

1.08 

2.46 

1.40 

Alt  2 

982 

1.45 

2.36 

1.52 

Alt  3 

1344 

1.81 

2.78 

1.63 

The  final  step  is  to  combine  the  decision 
maker's  utility  with  the  probability  distributions.  This 
is  done  by  integrating  the  multivariate  utility  function 
with  the  probability  distributions  to  yield  the  expected 
utility  for  each  alternative.  The  alternatives  are  then 
ranked  by  their  expected  utility.  Because  of  the 
assumptions  of  independence  and  risk  neutrality  the 
expected  utility  of  an  alternative  may  be  computed  by 
substituting  the  appropriate  mean  values  from  Table 
16  into  the  utility  function  U(c,x,y,z).  Table  17  gives 
the  expected  utilities  for  each  alternative. 

Table  17 

Expected  Utilities 


Alternative  Expected  Utility 


Base  Case  0.483 

Alt  1  0.428 

Alt  2  0.447 

Alt  3  0.465 


Now  alternative  three  ranks  first  followed  by 
the  base  case.  A  simple  computation  shows  that  with 
KC  valued  at  0.172,  the  base  case  and  alternative 
three  tie  for  the  first  rank  with  equal  expected 
utilities. 


Conclusion.  DoD  guidance  from  5000. 2M, 
Part  8,  on  the  use  of  integration  techniques  in  COEA 
includes: 


Analysis  of  Alternatives.  There  is  no  magic 
formula  for  combining  cost  and  effectiveness 
measures  to  identify  a  preferred  alternative. 
Judgments  and  perceptions  about  the  relative 
importance  of  competing  needs  are  important  in  the 
final  choice  of  a  course  of  action.  A  cost  and 
operational  effectiveness  analysis  can  assist  in  making 
that  choice  by  providing  a  solid  framework  for 
evaluating  the  alternatives,  and  by  highlighting  the 
implications  of  alternative  choices.  In  that  regard,  it 
is  essential  to: 


As  was  the  situation  with  AHP  the  base  case 
ranks  first  followed  by  alternative  three.  If  the  value 
of  KC,  which  equals  U(200,0. 25,0. 25,0. 25),  were 
decreased  from  0.20  to  0.15,  the  expected  utilities  of 
Table  17  would  be  replaced  by  those  of  Table  18. 


Table  18 

Revised  Expected  Utilities 


Alternative  Expected  Utilities 


Base  Case  0.451 

Alt  1  0.413 

Alt  2  0.435 

Alt  3  0.466 


Never  use  schemes  in  which  several 
measures  of  effectiveness  are  weighted  and  combined 
into  an  overall  score.  Weighting  schemes  can 
sometimes  be  helpful,  but  they  must  be  clearly 
explained  in  the  analysis  so  that  their  results  can  be 
interpreted  correctly. 

While  there  is  no  "magic"  formula  there  are 
formulas  that  can  capture  a  decision  maker's 
judgments  and  perceptions  for  the  relative  importance 
of  conflicting  attributes.  These  formulas,  or 
techniques,  may  be  based  on  reasonable  assumptions 
and  may  be  relatively  easy  to  apply.  All  practical 
techniques,  however,  require  some  degree  of 
participation  by  the  decision  maker.  Whether  this 
participation  is  to  be  accomplished  as  part  of  the 
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COEA  or  subsequent  to  it  becomes  an  important 
operational  question. 
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2.1.8  Managing  COEA  Development 


MANAGING  A  COEA  IS  NOT  UNLIKE  RIDING 
THE  LUGE.  THERE  IS  OFTEN  A  SENSATION 
OF  ROARING  DOWN  AN  ICY  MOUNTAIN.... 
PRONE  AND  SPREAD  EAGLE! 


Anonymous  Marine  Officer 


AAA  OVERALL 
STUDY  PROCESS 


•  COEA  ANALYZED  13  ALTERNATIVES 

AMPHIBIOUS  VEHICLES 
FAST  AND  SLOW 
NEW  AND  EXISTING 
NON-AMPHIBIOUS  VEHICLES 
NEW  AND  EXISTING 
NON-VEHICLES 
HELICOPTERS 
SUBMERSIBLES 

•  TWO  PHASED  STUDY  APPROACH 

PERFORMANCE  ANALYSIS 
OPERATIONAL  EFFECTIVENESS  ANALYSIS 


MISSION  AREA  ANALYSIS 


•  WHERE  THE  NEED  FOR  A  COEA  REALLY  BEGINS 

•  BECOME  FAMILIAR  WITH 

THE  THREAT  USED 
THE  MODEL(S)  USED 
THE  DOCTRINE  &  TACTICS  USED 
THE  EQUIPMENT  ANALYZED 
THE  DEFICIENCIES  IDENTIFIED 

•  DEFICIENCIES  BECOME  REQUIREMENTS 


THE  PURPOSE  OF  THE  COEA  IS  TO  FIND 
THE  BEST  WAY(S)  TO  RESOLVE  THEM 


PRE-MILESTONE-O 
"TRADE-OFF”  PROCESS 


•  THOROUGHLY  UNDERSTAND  WHY: 

CHANGES  IN  DOCTRINE,  TACTICS,  TRAINING 
OR  ORGANIZATION  DIDN’T  WORK. 

•  READ  EVERY  STUDY  OR  PAPER  ASSOCIATED 
WITH  THE  INVESTIGATION. 


•  REASON: 

GIVES  YOU  INSIGHT  INTO  THE  THREAT  AND 
HOW  YOU  MUST  "FIGHT’  THE  SYSTEM. 


I 
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YOU  MUST  HAVE 


•  MISSION  NEED  STATEMENT 

-  BEGINS  TO  DEFINE  THE  "UNIVERSE” 
OF  POTENTIAL  ALTERNATIVES 

•  JROC  PRESENTATION  AND  ENDORSEMENT 

-  THRESHOLDS  AND  GOALS 

•  ACQUISITION  DECISION  MEMORANDUM 

-  EXIT  CRITERIA 

•  IN  EACH  CASE; 

-  GO  REVIEW  THE  DOCUMENTS  WITH 
THE  PEOPLE  WHO  WROTE  THEM. 


STEP  ONE:  IDENTIFY  THE 
MAJOR  ELEMENTS  OF  A  COEA 

•  POLICY  {OSD  AND  SERVICE) 

•  THREAT  DATA 

•  ALTERNATIVE  CONCEPTS 

•  DOCTRINE  &  TACTICS  TO  BE  USED 

•  SCENARIOS 

•  ASSUMPTIONS 

•  MODELS 


METHODOLOGY  &  STUDY  PLAN 
COST  ESTIMATES 
■INTERESTED  PARTIES” 
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STEP  TWO:  ORGANIZE  THE 
SENIOR  LEVEL  LEADERSHIP 


•  ESTABLISH  A  GENERAL/FLAG  LEVEL 
ADVISORY  BOARD  VIA  YOUR  SPONSOR 

•  CRITICAL  TO  GETTING  RESOURCES 
AND  AUTHORITY  TO  DO  THE  JOB 

•  SERVES  AS  THE  QUALITY  CONTROL 
CHECK  YOU  WILL  NEED 

•  SAVES  A  LOT  OF  HEARTACHE  LATER 
IF  THEY  PROGRESS  ALONG  WITH  YOU 

•  PREVENTS  MISINTERPRETATION  OF 
DATA  AND  PROMOTES  CONSISTENCY 
IN  THE  SERVICE  POSITION 


NEW  DON  POLICY 
OVERSIGHT  BOARD 


•  PROGRAM  SPECIFIC 

•  NOT  A  FINAL  DECISION  FORUM 

RAISES/FORMS  ISSUES  FOR  ASN 
(RD&A)/CMC/CNO  DECISION 
WHEN  CONCENSUS  NOT  REACHED 

•  BOARD  REVIEWS: 

COMPLIANCE  WITH  OSD  GUIDANCE 
ASSUMTIONS  (VALID/COMPLETE) 
ALTERNATIVES,  MOEs,  THREAT, 
SCENARIOS,  ETC. 

•  OVERALL  QUALITY  CONTROL: 

DOES  IT  MAKE  SENSE? 

ARE  CONCLUSIONS  REASONABLE? 


STEP  THREE:  ORGANIZE  YOUR 

WORKING  LEVEL  RESOURCES 


•  BASED  ON  WHAT  THE  MAJOR  ELEMENTS  IMPLY 

•  IF  YOU  DON'T  HAVE  THE  AUTHORITY  TO  TASK 
AND  PRIORITIZE.... 

GO  GET  IT 

•  IDENTIFY  WHAT  COMMANDS,  AGENCIES,  LABS, 
ETC.  YOU  NEED  TO  ACCOMPLISH  THE  MAJOR 
ELEMENTS  OF  A  COEA 

GET  A  WORKING  LEVEL  REP  FROM  EACH 
OBTAIN  A  WRITTEN  MISSION  FOR  THE  GROUF 

CALL  IT  A  STUDY  ADVISORY  GROUP . 

IF  YOU  LIKE 


NEW  DON  POLICY: 
STUDY  DIRECTOR/TEAM 


•  STUDY  DIRECTOR 

PLANS  AND  SUPERVISES  COEA 
COORDINATES  FUNDING 
ESTABLISHES  TEAM  MEMBERSHIP 

•  MUST  HAVE 

ANALYSIS  BACKGROUND 

TECHNICAL  &  OPERATIONAL  BACKGROUND 

BE  INDEPENDENT  FROM  PMs 

•  STUDY  TEAM  GENERALLY  INCLUDES  WORKING 
LEVEL  REPS  OF  OVERSIGHT  BOARD  MEMBERS 

•  DIRECT  LIAISON  WITH  OSD  IS  AUTHORIZED 
FOR  TECHNICAL  DISCUSSIONS 


AAA  COEA  MANAGEMENT 
ORGANIZATION 


ESTABLISHED  CHARTERED  TASKED 

FLAG  OFFICER  REV  GRP  STUDY  ADVISORY  GRP  ANALYSIS  AGENCY 


DIR,  OPNs  DIV  (HQMC) 
DC/S  RQMT/PRG  (HQMC) 
CG,  MCRDAC 
DIR,  WARFIGHTING  CTR 
DACNO, SURFACE  WARFARE 
ASN(RDSA)  EXP  FORCES 
DR PM- AAA 


REPS  FOR  EA  FLAG  OFF 
PLUS- 

REP  OASD(PA&E) 

REP  OUSDA(A)  CENTER  FOR  NAVAL 

TECH  ADVISORS:  ANALYSIS 


ENGR  (DTRC) 
INTEL  (NTIC) 


ANALYSIS  PLAN 


GUIDANCE 


ANALYSIS  PRODUCTS 


REPORTING 


GUIDANCE  REVIEW 


MAJOR  ACTIVITIES  OF  THE 
AAA  SAG 


•  DEVELOP  INITIAL  STAR  THRU  INTEL  REP 

•  SELECT  SPECIFIC  SCENARIOS  THRU  USER  REP 

•  IDENTIFY  SYSTEM  ALTERNATIVES  THRU  SAG 

•  DEVELOP  ALTERNATIVES  'DESIGN'  BOOK 

THRU  ENGINEERING  AND  PM  REPS 

•  CONDITIONALLY  DEVELOP/ APPROVE  STUDY 

PLAN  &  METHODOLOGY... 

ASSUMTIONS 

MOEs 

MODELS 


PHASE  I:  PERFORMANCE  ANALYSIS 


•  PURPOSE  TO  SCREEN  MORE  CAPABLE  SYSTEMS 

FROM  THOSE  LESS  CAPABLE: 

GUIDES  THE  USE  OF  RESOURCES  ON  THE 
MOST  REASONABLE  ALTERNATIVES 

*  EACH  CANDIDATE'S  PERFORMANCE  WAS  ANALYZED 
BASED  ON  THEIR 

SHIP-TO-SHORE  MOVEMENT;  I.E. 

BUILD-UP  RATE  OF  COMBAT  POWER  ASHORE 
MOBILITY  ASHORE  IN  VARIOUS  TERRAINS 
SLOPE  CLIMBING 
MAXIMUM  SPEED 
VCI 

DRAW  BAR  PULL 
SURVIVABILITY 

PROBABILITY  OF  BEING  HIT 
PROBABILITY  OF  BEING  DAMAGED 
LETHALITY 

ACCURACY  AND  ARMOR  PENETRATION 


PHASE  II;  EFFECTIVENESS  ANALYSIS 


•  PURPOSE:  TO  SCREEN  SYSTEMS  AND  GUIDE  USE  OF 

RESOURCES  ON  MOST  LIKELY  ALTERNATIVES 

•  CONDUCTED  ON  7  OF  13  ALTERNATIVES 

•  USED  TWO  DIFFERENT  SCENARIOS 

LOW/MID  INTENSITY  COMBAT 
MID/HIGH  INTENSITY  COMBAT 
NO  RUSSIANS 

•  USED  TWO  DIFFERENT  SIZED  UNITS 

•  PRINCIPAL  MOE’s 

LOSS  EXCHANGE  RATIOS 
FORCE  MOVEMENT 
PERCENT  OF  SURVIVING  FORCE 
FORCE  RATIO 

SHIP-TO-SHORE  MOVEMENT  TIMED 

•  INCLUDED  SENSITIVITY  ANALYSIS 

ON  SEA  MINE  THREAT 
ENEMY  ARRIVAL  TIMES 

•  COSTED  ONLY  7  UNSCREENED  ALTERNATIVES 


MANAGING  A  COEA 
LESSONS  LEARNED 


•  DESIGN  A  PROCESS  THAT  CAN  ACCOMMODATE  CHANGES 

•  YOU  CANT  START  TOO  EARLY  TO  DEVELOP/VALIDATE 
THE  THREAT 

•  MAKE  SURE  YOUR  SCENARIO  THREATS  (EQUIPMENT, 

OOB,  DOCTRINE,  ETC)  ARE  IN  THE  STAR 

•  MAKE  SURE  YOUR  ASSUMPTIONS  ARE  LOGICAL,  DEPENDABLE, 
WELL  DOCUMENTED  AND  AGREED  TO  BY  EVERYONE 

•  ESTABLISH  AN  EARLY  RELATIONSHIP  BETWEEN  ALL  THE 
PM-’COSTERS"  AND  "ICE  PEOPLE’  IN  THE  CAIG 

•  DON'T  ’COOK  THE  BOOKS’  ON  SCENARIOS  -  PARALLEL  THE 
REAL  WORLD,  CONSIDER  USING  MORE  THAN  ONE,  AND  APPLY 
THE  ’MOST  LIKELY’  LITMUS  TEST. 

•  PICK  THE  RIGHT  KIND  OF  MODEL,  UNDERSTAND  ITS 
LIMITATIONS  AND  GET  A  ’SEAL  OF  APPROVAL’  FOR  IT 

•  PICK  MOE'S  THAT  HAVE  MEANING  TO  WARFIGHTERS 


MANAGING  A  COEA 
LESSONS  LEARNED  (CONT) 


•  KEEP  THE  SAG  SMALL,  ENSURE  THAT 
IT  HAS  AUTHORITY  TO  MAKE  ROUTINE 
DECISIONS,  INCLUDE  OSD  REPS 

•  BE  EVENT  ORIENTED  WHEN  CALLING  FOR 
A  SAG  MEETING 

•  DOCUMENT  EVERYTHING  SAID  OR  DONE 

•  KEEP  ALL  YOUR  LEADERSHIP  INFORMED 
AND  UP  TO  DATE 

•  DON’T  WAIT  UNTIL  THE  END  TO  PUBLISH 
RESULTS 


DON’T  EXPECT  THE  COEA  TO  PROVIDE 
'THE  ANSWER’  _ 


2.1.9  Linking  COEAs  to  OT&E 


MORS  COEA  SYMPOSIUM 


LINKING 
COEA  TO  OT&E 


Dr  Bill  Lese 
OASD(PA&E) 


ACQUISITION  POLICY 


•  STATES  THAT  BOTH  COST  AND  OPERATIONAL 
EFFECTIVENESS  ANALYSES  AND  TEST  AND 
EVALUATION  ARE  AIDS  TO  DECSIONMAKING 

ALSO 

♦  STATES  THAT  LINKAGE  SHOULD  EXIST 
BETWEEN  COEAs  AND  TEST  AND  EVALUATION 
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ACQUSITION  POLICY 

DoDI  5000.2/4/E/3.a(5)(c) 


MEASURES  OF  EFFECTIVENESS  SHOULD  BE 
DEVELOPED  TO  A  LEVEL  OF  SPECIFICITY 
SUCH  THAT  A  SYSTEM’S  EFFECTIVENESS 
DURING  DEVELOPMENTAL  AND  OPERATIONAI 
TESTING  CAN  BE  ASSESSED  WITH  THE  SAME 
EFFECTIVENESS  CRITERIA  AS  USED  IN  THE 
COST  AND  OPERATIONAL  EFFECTIVENESS 
ANALYSIS.  THIS  WILL  PERMIT  FURTHER 
REFINEMENT  OF  THE  ANALYSIS  TO  REASSESS 
COST  EFFECTIVENESS  COMPARED  TO  ALTERNA 
TIVES  IN  THE  EVENT  THAT  PERFORMANCE  AS 
DETERMINED  DURING  TESTING,  INDICATES  A 
SIGNIFICANT  DROP  IN  EFFECTIVENESS  (I.E., 

TO  OR  BELOW  A  THRESHOLD)  COMPARED  TO  THE 
LEVELS  ASSUMED  IN  THE  INITIAL  ANALYSIS. 


ACQUSITION  POLICY 
DoDI  5000.2/4/E/3.a(5) 


TO  JUDGE  WHETHER  AN  ALTERNATIVE  IS 
WORTHWHILE,  ONE  MUST  FIRST  DETERMINE 
WHAT  IT  TAKES  TO  MAKE  A  DIFFERENCE. 
MEASURES  OF  EFFECTIVENESS  SHOULD  BE 
DEFINED  TO  MEASURE  OPERATIONAL  CAPA¬ 
BILITIES  IN  TERMS  OF  ENGAGEMENT  OR 
BATTLE  OUTCOMES.  MEASURES  OF  PERFOR¬ 
MANCE,  SUCH  AS  WEIGHT  AND  SPEED,  SHOULD 
RELATE  TO  THE  MEASURES  OF  EFFECTIVENESS 
SUCH  THAT  THE  EFFECT  OF  A  CHANGE  IN  THE 
MEASURE  OF  PERFORMANCE  CAN  BE  RELATED 
TO  A  CHANGE  IN  THE  MEASURE  OF  EFFECTIVE¬ 
NESS... 


ACQUSITION  POLICY 

DoDI  5000.2-M/8/2.a(5) 


A  COMPREHENSIVE  TEST  AND  EVALUATION 
PROGRAM  IS  AN  INTEGRAL  FACTOR  IN 
ANALYZING  OPERATIONAL  EFFECTIVENESS, 
SINCE  IT  WILL  PROVIDE  TEST  RESULTS  AT 
EACH  MILESTONE  DECISION  POINT  THAT 
GIVE  CREDENCE  TO  THE  KEY  ASSUMPTIONS 
AND  ESTIMATES  THAT  MAY  HAVE  BEEN  MADE 
IN  THE  CURRENT  OR  EARLIER  COST  AND 
OPERATIONAL  EFFECTIVENESS  ANALYSES. 


COEA  -  T&E  RELATIONSHIP 


THE  LINKAGE  BETWEEN  COEA  AND  T&E 
IS  COMPLICATED  BY: 

•  COEA  AND  T&E  BEING  EXECUTED  BY 
DIFFERENT  AGENCIES. 

•  MOE  AND  MOP  BEING  DEVELOPED  INDEPENDENTLY 

•  THE  OT  ENVIRONMENT  DIFFERING  SIGNIFICANTLY 
FROM  THAT  ASSUMED  IN  THE  COEA. 

•THE  COEA  BEING  "PUT  ON  THE  SHELF"  AT  MS  II 
AND  THE  RATIONALE  FOR  DECIDING  WHETHER  A 
SYSTEM  WAS  A  COST-EFFECTIVE  APPROACH  TO 
MEETING  THE  OPERATIONAL  REQUIREMENT  BEING 
FORGOTTEN. 
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COEA  -  T&E  RELATIONSHIP 

(CONTINUED) 


THE  REMEDIES  LIE  IN: 

EARLY  COORDINATION  BETWEEN  THE  AGENCIES 

*  RESPONSIBLE  FOR  THE  COEA,  T&E,  AND  THE 
DEVELOPMENT  OF  REQUIREMENTS. 

IDENTIFICATION  OF  MOEs  FOR  BOTH  COEA  AND 

*  T&  E  THAT  ARE  DERIVED  FROM  THE  MISSION 
NEED  STATEMENT  AND  REFLECT  OPERATIONAL 
UTILITY  AND  A  CLEAR  DEMONSTRATION  OF  THE 
RELATIONSHIP  BETWEEN  MOEs  AND  MOPs. 

UNDERSTANDING  OF  THE  REASON  FOR  AND 

*  MAGNITUDE  OF  THE  DIFFERENCE  ANTICIPATED 
BETWEEN  TEST  AND  ANALYTIC  RESULTS. 

ABILITY  TO  REVIEW  THE  COEA  USING  TEST 

*  RESULTS  TO  REAFFIRM  EARLIER  DECISIONS. 


COEA  -  T&E  RELATIONSHIP 

GUIDELINES 


•  DoD  COMPONENTS,  IN  THE  PROCESS  OF  PERFORMING 
MS-1  COEA  SHOULD  IDENTIFY  THE  MOEs  TO  BE 
USED  IN  THE  COEA  AND  SHOW  HOW  THESE  MOEs  ARE 
DERIVED  FROM  THE  MNS. 

•  COEA  SHOULD  INCLUDE  MOEs  REFLECTING  OPERA¬ 
TIONAL  UTILITY  THAT  CAN  BE  TESTED.  THESE  AND 
OTHER  MOEs  THAT  CANNOT  BE  TESTED  SHOULD  BE 
DESCRIBED  IN  TERMS  OF  TESTABLE  PARAMETERS  OR 
MOPs  (I.E.  SHOW  HOW  CHANGES  IN  THESE  PARAMETERS 
RELATE  TO  CHANGES  IN  THE  COEA  MOEs.) 

•  THESE  MOEs,  MOPs,  AND  THE  ASSOCIATED  CRITERIA 
SHOULD  BE  INCLUDED  IN  THE  ORD,  AND  THE  KEY 
MOEs  SHOULD  ALSO  BE  INCLUDED  IN  THE  APB. 

•  CONSISTENCY  SHOULD  BE  MAINTAINED  BETWEEN  ALL 
THE  ACQUISITION  MANAGEMENT  DOCUMENTATION. 
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COEA  -  T&E  RELATIONSHIP 
GUIDELINES  (CONT) 


THERE  SHOULD  BE  THE  ABILITY  TO  REVIEW  THE 

COEA  AT  MILESTONE  DECISON  POINTS  AND  OTHER 
ACQUISITION  REVIEWS  USING  TEST  RESULTS 
(DEVELOPMENTAL/OPERATIONAL  AS  APPROPRIATE) 
TO  REAFFIRM  THE  DECISIONS  MADE  EARLIER 

SELECTED  ALTERNATIVE  IS  STILL  COST 

•  EFFECTIVE 

OPERATIONAL  REQUIREMENTS  ARE  MET 

IF  THE  ORIGINAL  COEA  ASSUMPTIONS  ARE  NOT 
SUPPORTED  BY  TEST  RESULTS  OF  THE  THRESHOLDS 
IN  THE  ORD  ARE  NOT  MET,  COEA  SENSITIVITY 
ANALYSES  SHOULD  BE  AVAILABLE  TO  ASSIST  THE 
DECISIONMAKERS  IN  DETERMINING  WHETHER 

•  THE  SYSTEM  IS  STILL  COST  EFFECTIVE 

•  DOT&E  IS  ABLE  TO  CONFIRM  THAT  THE 
SYSTEM  IS  OPERATIONALLY  EFFECTIVE  AND 
SUITABLE 


COEA  -  T&E  RELATIONSHIP 
GUIDELINES  (CONT) 


FOR  A  VARIETY  OF  REASONS  (EG.  AFFORDA¬ 
BILITY,  AVAILABILITY  OF  TEST  RESOURCES, 

SAFETY  CONSTRAINTS)  THE  PROPOSED  OPERA¬ 
TIONAL  ENVIRONMENT  OFTEN  DIFFERS  SIGNI¬ 
FICANTLY  FROM  THAT  ASSUMED  IN  THE  COEA 

•  PROPOSED  OPERATIONAL  SCENARIO  AND 
THREAT  REPRESENTATION 

•  SAFETY  AND  OPERATING  RESTRICTIONS 

•  DIFFERENCE  BETWEEN  TEST  ARTICLE  AND 
THE  SYSTEM  AS  REPRESENTED  IN  THE  COEA 

(EG.  MATURITY  &  LEVEL  OF  MATURITY) 

IF  AS  A  CONSEQUENCE,  TEST  RESULTS  ARE 
EXPECTED  TO  SIGNIFICANTLY  DIFFER  FROM 
COEA  ASSUMPTIONS  ,  THE  COEA  SHOULD  ASSESS 
THE  MAGNITUDE  OF  THE  IMPACT  AND  THE  EXTENT 
TO  WHICH  TEST  AND  ANALYTIC  RESULTS  WOULD 
BE  EXPECTED  TO  VARY 
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MORS  COEA  SYMPOSIUM 


LINKING 
COEA  TO  T&E 


Mr  Richard  R.  Ledesma 
OUSD(A)  (T&E) 


COEA  RELEVANCE  TO 
TEST  &  EVALUATION 


AN  ANALYSIS  TOOL 

PROVIDES  SYSTEM  FOCUS  ON  THE  BASIS  OF 
REQUIREMENTS  INCLUDING  FEASIBILITY  TO 
MEASURE  PERFORMANCE  AND  THREAT 

SYNTHESIZES  REQUIREMENTS,  THREAT  AND 
COST  TO  FORM  A  BASIS  FOR  A  DT  &  OT 
PERFORMANCE  PARAMETERS. 

NOT  AN  EVALUATION  TOOL  -  BUT  PROVIDES 
A  FRAMEWORK  FOR  T&E 
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DT  CONTRIBUTION  TO 
T&E  -  COEA  LINKAGE 


•  ASSISTS  COEA  INI  DERIVING  APPROPRIATE  MOE/MOP 

•  ESTABLISHES  FEASIBILITY  OF  ACHIEVING  MOE/MOP 

**  MEASURABLE 

**  ACHIEVABLE  (I.E.  EXCHANGE  RATIO) 

•  PROVIDES  T&E  RESULTS  INPUT  TO  LATER  COEA 
QUANTIFIES  MEASURE  OF  TECHNICAL  PERFORMANCE 

•  CERTIFIES  MODELING  &  SIMULATION  AS  TESTING 
ADJUNCT 

•  PROVIDES  UPDATES  &  MEANS  TO  REFINE  MOE/MOP 
DURING  DEVELOPMENT 

•  PROVIDES  EARLY  INSIGHT  INTO  ACHIEVABLE 
OPERATIONAL  PERFORMANCE 
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DT/COEA  RESOURCE 
IMPLICATIONS 


•  EARLY  INTERACTION  TO  ADDRESS  TESTING  AND 
RESOURCE  ISSUES. 

9  WHAT  IF  RESOURCES  ARE  NOT  AVAILABLE  ? 

**  DON’T  PROCEED? 

**  ACCEPT  INADEQUATE  TESTS? 

**  LOOK  FOR  ALTERNATIVES? 

•  WHEN?  BETWEEN  MS-0  AND  MS-1 


COEAfTAE  LINKAGE  COULD 
PROVIDE  FIRST  OPPORTUNITY 
TO  ADDRESS  THIS  IISUE 
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2.1.10  The  COEA  In  Support  of  the  DoD  Decision 
Process 


Introduction 

Cost  and  Operational  Effectiveness  Analysis 
(COEA)  and,  more  generally,  systems  analysis  are  well 
established  methodologies  used  by  the  Department  of 
Defense  (DoD)  to  allocate  scarce  resources.  The 
methodologies  were  introduced  into  the  decision-making 
process  along  with  the  "systematic"  Planning, 
Programming  and  Budgeting  System  (PPBS)  in  the  early 
sixties.  Yet  while  the  PPBS  in  DoD  has  been  from  its 
earliest  days  generally  well  received,  systems  analysis, 
including  COEAs,  is  often  highly  controversial.  COEAs 
are  alternately  viewed  with  hostility  or  favor.  When  their 
results  support  a  "defense  consensus,"  COEAs  sail 
through  the  bureaucracy.  But  woe  the  COEA  that 
challenges  an  "obvious"  defense  need  or  supports  a 
"turkey"  of  a  system.  It  is  puzzling,  indeed,  that  an 
enterprise  whose  fundamental  strength  is  "objectivity" 
can  evoke  such  reactions.  Yet,  in  spite  of  several 
attempts  over  the  years  to  diminish  the  role  of  analysis, 
the  Office  of  the  Secretary  Of  Defense  (OSD)  continues 
to  place  high  emphasis  on  COEAs  and  other  analytical 
studies.  Guided  by  the  principles  that  decision-making 
can  be  improved  by  analysis  and  that  it  is  always 
desirable  to  examine  the  cost-effectiveness  of  alternatives 
to  meet  requirements,  the  DoD  has  integrated  COEAs 
into  its  new  acquisition  regulations  (the  so  called  5000 
series  regs).  From  the  DoD  perspective,  a  COEA  always 
serves  a  valuable  function  for  even  when  it  does  not 
definitively  resolve  all  issues,  it  promotes  a  logical 
presentation  of  information  and  provides  a  rational  basis 
for  evaluating  alternative  courses  of  action.  Because  a 
COEA  underpins  very  difficult  decisions  with  analysis,  it 
will  continue  to  play  a  significant  role  in  the  OSD 
decision-making  machinery. 

The  OSD  and  the  Services  have  developed 
elaborate  organizations  of  people,  models  and 
methodologies  to  do  COEAs,  making  considerable 
investments  in  studies  for  each  cycle  of  acquisition  and 
for  program  reviews.  The  Army,  for  example,  the  service 
most  heavily  invested  in  analysis,  not  only  develops  each 
year  a  comprehensive  study  program  (the  AR  5-5  Study 
Program)  but  also  maintains  entire  organizations  devoted 
to  executing  it.  The  Navy,  on  the  other  hand,  where 
analysis  traditionally  has  been  dispersed  and  more 
"targeted,"  is  currently  setting  up  an  organizational 
structure  to  integrate  the  COEA  "process”  into  overall 
decision  making.  The  Air  Force  is  likewise  revising  its 


analysis  structure,  reallocating  responsibility  for  COEAs 
to  the  major  commands  where  the  studies  can  be 
appropriately  integrated  into 

Air  Force  decisions.  OSD,  of  course,  depends  mainly  on 
"outside"  or  contractor  support  for  its  studies  but  it  is  no 
less  committed  to  the  COEA  process.  In  all  cases,  money 
is  being  allocated  to  support  new  study  initiatives.  Given 
that,  a  COEA,  as  a  rule  of  thumb,  takes  from  six  to  twelve 
man-years  and  costs  between  two  and  five  million  dollars, 
the  continued  reliance  by  all  parties  on  quantitative 
evaluations  is  a  clear  indication  of  the  value  attached  to 
such  work. 

It  is  worth  noting  that  COEAs  ideally  should  be 
neither  shields  nor  swords  in  the  fight  for  defense 
resources  but  rather  merely  tools  for  enlightened 
decision-making.  In  their  pure  essence,  they  are  most 
useful  and  productive  when  helping  decision-makers  to 
formulate  problems,  choose  appropriate  objectives  and 
alternatives,  compare  and  test  alternatives  in  realistic 
environments  and  assist  in  setting  priorities  and  reaching 
decisions.  COEAs  do  not  "give  the  best  solution"  but 
provide  the  vehicle  for  decision-makers  to  make  tough 
choices  with  objective  clarity. 

Why  Focus  on  COEAs 

COEAs  are  important  for  two  reasons:  they  are 
the  primary  tools  for  resolving  resource  allocation  issues 
concerning  the  discretionary  part  of  the  DoD  budget;  and 
they  provide  analytical  justification  for  selected  courses 
of  action. 

It  has  been  argued  (Hitch)  that  the  basic 
problems  of  defense  are  economic.  That  is  to  say,  most 
defense  issues  eventually  devolve  to  a  choice  of 
alternative  means  for  satisfying  objectives  within  cost  or 
other  resource  constraints.  It  is  not  uncommon,  therefore, 
that  defense  problems  spill  over  to  the  general  federal 
budget  arena  since,  as  defense  competes  within  the 
broader  set  of  federal  expenditures,  choices  made  in 
defense  can  impact  the  nation's  general  economy  and 
vice-versa  Moreover,  the  impact  of  defense  choices  can 
in  fact  be  larger  than  just  the  magnitude  of  defense 
expenditures  themselves  (nominally  about  25-30%  of  the 
Federal  budget)  because  a  large  fraction  (nearly  68%  and 
growing  as  apparent  in  Figure  1)  of  the  federal  budget  is 
"mandatory",  i.e.  non-discretionaiy,  spending. 


79 


FEDERAL  BUDGET  OUTLAYS 


I9ILUON 
tCOO  , 


I 


Figure  1 


Hence  in  the  fight  for  resources,  the  pie  over 
which  the  national  priorities  battle  takes  place  is  mostly 
defense  is  fact,  nearly  70%  of  federal  discretionary 
funds  ar,  Defense  dollars  (Figure  2). 
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Defense  spending,  as  viewed  from  the  federal 
budget  perspective,  is  all  discretionary. 

There  is  no  denying  that  in  both  relative  and 
absolute  terms,  the  amount  of  money  authorized  for 
defense  is  huge  (Figure  3). 
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Figure  3 


However,  when  viewed  strictly  from  a  defense 
perspective,  the  entire  defense  budget  is  not 
discretionary.  For  all  intents  and  purposes,  once  a  force 
structure  is  established,  the  money  to  support  that  force 
itself  becomes  mandatory,  covering  such  items  as 
personnel  pay,  quality  of  life  and  operations  and 
maintenance.  None  of  these  can  be  "raided"  with 
impunity  without  serious  consequence  for  total  force 
capability.  Furthermore,  in  recent  years,  the  DoD  has 
placed  great  emphasis  on  avoiding  a  "hollow"  force, 
meaning  that  funds  for  these  three  "mandatory"  areas 
enjoy  high  priority.  The  current  Annual  Report  of  the 
Secretary  of  Defense  makes  this  clear  (page  24),  noting 
that  defense  priorities  place  people,  quality  of  force  and 
readiness  at  the  top  of  the  list.  The  net  result  is  that  the 
discretionary  portion  of  the  defense  budget  essentially 
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defaults  to  the  Research  and  Development  (R&D)  and 
Procurement  accounts.  An  additional  small  amount, 
about  3%  of  the  total  DoD  budget,  allocated  for  military 
construction,  can  also  be  included  in  this  pot.  Together 
these  moneys,  are  referred  to  as  the  "investment" 
account.  In  recent  years,  this  account  has  averaged 
about  35%  or  $95  Billion,  no  small  change  to  be  sure. 

In  effect,  therefore,  the  basic  economic 
problem  for  defense  is  how  to  allocate  DoD  investment 
account  funds  or  how  best  to  invest  in  modernization 
and  improvement,  (i.e.  acquisitions)  for  our  forces. 
Considerable  effort  has  been  expended  by  the  DoD  in 
recent  months  to  address  this  problem.  General 
initiatives  to  improve  the  acquisition  process  have  been 
well-publicized.  But  a  more  technical  initiative  has 
also  been  launched  with  the  publication  and 
implementation  of  the  5000  series  acquisition 
regulations.  Included  in  these  regulations  as  an  integral 
and  recurring  part  of  the  new  process  is  the  COEA. 
According  to  these  regulations,  a  COEA  provides  the 
only  comparative  analysis  for  evaluating  alternative 
acquisitions  and  is  a  key  document  for  establishing 
alternative  preferences.  The  COEA  performs  three 
functions  as  part  of  this  process.  It; 

1)  aids  decision  makers;  it  presents  and 
uncovers  relative  advantages  and  disadvantages  of 
alternatives  and  provides  a  measure  of  the  relative 
sensitivity  of  the  alternatives  to  changes  in  underlying 
assumptions; 

2)  facilitates  communications;  it  identifies 
feasible  alternatives  and  promotes  open  debate 
concerning  their  reasonableness,  efficacy  and  suitability 
for  given  objectives,  assumptions  and  limitations; 

3)  documents  acquisition  decisions;  it 
provides  the  historical  written  record  of  the  rationale  for 
selected  courses  of  action  so  it  can  be  reexamined  later, 
perhaps  under  different  circumstances  and  by  different 
players. 

The  acquisition  process  requires  numerous 
studies  and  voluminous  documentation.  It  is 
complicated,  time  consuming,  and  demanding.  No 
doubt,  this  drives  much  of  the  effort  to  underpin 
decisions  analytically.  However,  of  the  analyses  that 
are  required,  only  one  type  evaluates  alternative  courses 
of  action  in  terms  of  cost-effectiveness.  That  is  to  say, 
although  much  analysis  is  done  leading  up  to  the  point 
of  identifying  a  materiel  solution  to  meet  a  DoD 
requirement,  most  of  this  analysis  deals  with 


requirements  evaluation  or  with  the  execution  of  the 
acquisition  process  itself,  not  with  the  problem  of 
economic  choice.  The  COEA  alone  addresses  the 
matter  of  "military  worth"  vis-a-vis  cost. 

The  cost-effectiveness  tradeoff  is  far  from  a 
trivial  concern.  In  DoD,  the  fundamental  issue  is  how 
to  choose  doctrine,  weapons,  and/or  equipment  to  get 
the  most  "defense"  for  the  dollars  available.  The  issue 
is  not  how  to  maximize  effectiveness  or  how  to 
minimize  cost.  It  is  neither  a  chauvinistic  appeal  to 
getting  the  "best  for  our  boys"  nor  a  scrooge-like  denial 
to  make  do  with  the  worst  possible  systems.  It  a  matter 
of  weighing  the  value  or  utility  of  an  alternative  against 
the  cost  of  implementation.  This  problem  exists 
because  resources  are  always  limited  and  what  is  used 
for  one  purpose  decreases  the  amount  available  for 
another.  The  COEA  gives  decisionmakers  a  framework 
for  assessing  these  choices.  To  the  extent  that  it  is  the 
only  such  decision  support  tool  in  the  acquisition 
process,  the  COEA  is  essential. 

In  addition,  examining  the  new  acquisition 
process  reveals  that  only  one  type  of  analysis  recurs: 
the  COEA.  Requirement  studies  are  initiated  once, 
acquisition  and  implementation  plans  once  (and 
adjusted  thereafter);  and  concept  studies  once.  The 
COEA,  however,  is  done  or  updated  at  each  milestone 
of  the  process.  In  this  sense,  it  is  a  dynamic  evaluation 
of  choices  and  serves,  at  each  point,  to  confirm  that  the 
solution  being  pursued  remains  viable.  The  COEA, 
therefore,  is  the  thread  that  weaves  all  procurement 
justifications  together.  Consequently,  it  is  as  much 
forward  looking  as  it  is  backward  looking.  That  is,  it 
emphasizes  the  evaluation  of  current  alternatives  but  i: 
does  so  by  asking  whether  the  original  requirement  is 
still  valid.  For  this  reason,  sometimes  it  can  be  a 
"show-stopper"  but  always  serving  the  decision-maker 
by  asking  whether  "the  gain  is  still  worth  the  cost."  No 
responsible  official  would  fail  to  ask  that  question  at 
key  review  points  and  it  follows  that  the  COEA  would 
always  be  important  for  making  that  judgment.  In 
effect,  the  COEA  in  its  replications  provides  the 
continuous  justification  to  proceed  with  an  acquisition. 

What  is  a  COEA 

A  COEA  is,  first  and  foremost,  an  aid  to  the 
decision-maker.  It  provides  a  framework  for  comparing 
alternative  courses  of  action  using  a  structured 
quantitative  approach.  It  addresses  head  on  the  basic 
allocation  problem  of  defense  which,  in  simplest  terms, 
is  either  to  get  the  most  "defense"  for  a  given  level 
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resources  (i.  e.  the  classical  "bang  for  buck"  problem) 
or,  equivalently,  to  achieve  a  given  level  of  defense  at 
the  least  cost.  While  these  perspectives  are 
conceptually  easy  to  grasp,  they  are  difficult  to  execute. 
Frequently,  there  are  either  too  many  choices  or  the 
context  for  choice  is  extremely  complicated.  The 

COEA  helps  clarify  the  pros  and  cons  of  complex 
decisions  and  guides  the  decision-maker  in  asking  the 
right  questions  and  uncovering  uncertainty  by  steering 
his  thinking;  the  COEA  provides  the  decision-maker 
with  confidence  that  the  solution  is  a  product  of  a 
rational  decision  process. 

In  DoD,  a  COEA  is  defined  as: 

a  comparative  analysis  of  given 
alternatives  (one  of  which  is  a 
basecase)  intended  to  meet  specified 
requirements  in  an  operational 
context  and  using  comparative  life 
cycle  cost  estimates  to  establish 
alternative  preferences  based  on 
cost-benefit  measures. 

This  definition  embodies  several  key  concepts 
for  analysis.  First,  a  COEA  is  comparative  in  nature, 
implying  that  "relative  measurement"  is  the  desired 
analysis  framework.  In  particular,  this  means  that  the 
COEA  analyst  must  identify  and  highlight  the  critical 
distinguishing  characteristics  of  the  alternatives.  In 
general,  a  COEA  is  of  little  value  in  absolute  terms. 
This  is  a  distinguishing  feature  of  most  COEAs  and  is 
the  reason  that  COEA  comparisons  often  remain  valid 
even  as  underlying  assumptions  in  the  study  change. 
Unlike  "absolute"  analyses,  in  a  COEA  only  differences 
among  alternatives  are  ever  quantified.  Thus,  the 
critical  questions  when  assumptions  change  are  "does 
the  relative  ranking  of  the  alternatives  also  change"  and 
"is  the  delta  between  alternatives  now  small  enough  to 
reverse  the  cost-effectiveness  preferences"?  The 
answers  to  these  questions  are  frequently  in  the 
affirmative. 

Second,  the  typical  COEA  accepts  alternatives 
as  inputs.  In  other  words,  alternatives  are  given.  This 
does  not  preclude,  however,  the  consideration  of 
additional  alternatives  even  as  a  COEA  is  being 
executed.  Rather,  as  new  alternatives  are  identified, 
whether  prior  to  or  during  a  COEA,  they  must  be 
introduced  at  the  beginning  of  the  COEA  methodology. 
Normally,  the  COEA  methodology  will  not  give  rise  to 
conceptually  different  alternatives,  although  variations 
on  a  theme  frequently  do  arise.  The  definition  of  the 


alternative  set  is  made  outside  the  COEA  framework, 
sometimes  as  a  matter  of  policy  -  as  for  example  when 
a  foreign  system  is  (or  is  not)  decreed  to  be  considered 
in  the  analysis.  More  often,  however,  alternatives  enter 
naturally  as  the  decision  issues  are  developed.  Further, 
since  one  of  the  alternatives  must  be  the  basis  for 

comparison,  the  "basecase"  alternative,  (also  called  the 
"status  quo”  or  "default"),  must  be  included.  In  general, 
therefore,  alternatives  are  defined  prior  to  starting  a 
COEA.  But  as  with  all  external  COEA  parameters, 
inputs  can  drive  outputs.  So  caution  must  be  exercised 
to  avoid  excluding  potential  solutions  by  assumption. 
After  all,  the  point  of  doing  a  COEA  is  to  see  whether 
cost-effective  improvement  can  be  obtained  over  the 
status  quo.  An  alternative  set  that  is  not  broad  enough 
to  cover  the  decision  issues  defeats  that  purpose.  A 
COEA  that  is  short  on  alternatives  will  not  justify  the 
economic  value  of  substituting  something  new  for 
something  on-hand.  And,  it  certainly  will  not  aid  the 
decision-maker. 

Third,  a  requirement  for  a  materiel  acquisition  is 
presumed  to  have  been  established  prior  to  initiating  a 
COEA.  By  the  time  the  COEA  starts,  a  "Mission  Need 
Analysis  (MNA)"  should  have  been  completed.  Such 
an  analysis  should  have  established  that  a  specified 
requirement  can  be  met  only  by  a  materiel  solution.  A 
COEA  will  not  provide  a  basis  for  determining  whether 
a  requirement  is  valid  or  whether  a  deficiency  is  real. 
This  is  beyond  the  scope  of  the  methodology.  The 
COEA  will,  of  course,  raise  questions  about 
requirements  as  differences  in  alternatives  are 
quantified  and  rationalized.  In  this  respect,  the  COEA 
can  and  should  complement  early  conceptual  studies  of 
requirements  and,  more  importantly,  should  have  a  link 
back  to  them  so  that  any  such  questions  can  be  resolved 
as  part  of  the  "requirements  validation"  at  each 
acquisition  milestone. 

Probably  the  most  mature  concept  of  a  COEA,  as  it 
is  done  in  the  DoD,  is  that  of  operational  effectiveness 
analysis.  It  is  this  concept  that  distinguishes  DoD 
analysis  from  general  economic  analysis.  It  may  also 
be  the  reason  why  other  government  agencies  have  had 
difficult}-  in  applying  it  to  their  operations.  Notionally 
operational  effectiveness  says  that,  when  comparing 
alternatives,  the  issue  is  not  how  well  an  individual 
system  performs  but  rather  how  well  it  contributes  to 
mission  accomplishment.  The  former  perspective  is 
almost  inconsequential  to  a  COEA  (except  as  an  input) 
because  in  most  cases  from  this  perspective  the  answer 
is  usually  obvious:  new  is  better,  bigger  is  better, 
advanced  is  better.  However,  from  the  mission 
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completion  perspective,  the  issue  is  one  of  "force-level" 
effectiveness.  It  is  a  team  concept,  if  you  will,  where 
the  value  of  a  system  is  measured  in  terms  of  its 
integral  contribution  to  an  overall  effort  by  an 
organization.  Capability  that  contributes  to  this  goal  is 
of  value  while  that  which  does  not,  no  matter  how 
flashy  from  an  individual  perspective,  does  not  count. 
This  is  the  reason  that  DoD  has  spent  enormous  energy 
in  developing  combat  models  and  conducting 
operational  tests.  It  is  through  simulation,  modeling, 
and  testing  at  the  force-on-force  level  that  alternatives 
can  be  assessed  for  their  mission  accomplishment 
value.  A  properly  executed  COEA  will,  therefore, 
show  how  well  one  alternative  does  as  part  of  a  military 
team  conducting  realistic  missions  in  an  integrated, 
combined-arms  battlefield. 

It  is  an  acknowledged  fact  that  the  cost  of 
systems  exceeds  the  cost  of  acquiring  them  for  the  cost 
of  maintaining  and  supporting  systems  must  also  be 
included.  Moreover,  history  has  shown  that  these  latter 
costs  are  the  most  significant.  Taken  together,  the  cost 
of  acquisition  including  R&D,  and  operations  and 
support,  constitutes  the  life  cycle  cost  of  a  system  or  the 
entire  cost  of  "ownership."  The  only  proper  basis  for 
comparing  COEA  alternatives  therefore  is  Life  Cycle 
Cost  (LCC).  In  DoD,  the  typical  term  for  LCC 
estimates  is  20  years.  However,  in  order  to  compare 
alternatives,  it  is  not  sufficient  to  develop  only  20  year 
life  cycle  cost  estimates  for  each  one  for  there  would  be 
no  way  to  account  for  the  "phasing  in"  of  an  alternative. 
That  is  to  say,  there  is  a  cost  associated  with  each 
alternative,  beyond  its  life  cycle  cost,  for  bringing  it 
into  the  inventory.  Depending  on  the  alternative,  these 
costs  could  be  substantial  and  if  ignored  could  lead  to 
an  erroneous  indication  of  the  true  cost  of  an 
alternative.  In  order  to  get  a  true  comparative  cost 
basis,  all  COEA  alternatives  should  be  synchronized 
relative  to  their  phasing-in  schedules.  The  costs 
associated  with  phasing-in  plus  the  life  cycle  cost 
comprise  the  "comparative"  LCC,  the  proper  basis  for 
comparing  the  costs  of  COEA  alternatives. 

The  integration  of  cost  and  effectiveness  is  the 
last  key  concept  in  a  COEA  and  the  hardest  part  to 
execute.  Notionally,  the  idea  is  simple:  combine  the 
two  attributes,  cost  and  effectiveness,  into  a  single 
measure  -  perhaps  a  ratio  or  a  weighted  sum.  But  as  is 
well  known,  ratios  can  distort  differences  and  weighted 
averages  can  inject  bias.  So  while  the  problem  is  easy 
to  conceptualize  analytically,  it  is  a  difficult  decision- 
theoretic  problem  to  solve.  Many  techniques  have  been 
developed  to  address  this  problem,  including  cost- 


effectiveness  and  relative  worth  ratios,  multi-attribute 
utility  theory,  and  the  more  recent  pairwise  comparative 
methods  like  the  analytic  hierarchy  process. 
Nevertheless,  the  problem  of  identifying  alternative 
preferences  using  strictly  quantitative  methodologies 
remains  unsolved  though  considerable  research  effort 
continues. 

Analytical  difficulties  not  withstanding,  preference 
and  choice  are  still  the  objectives  in  a  COEA.  Thus,  if 
it  is  to  serve  its  purpose,  a  COEA  must  clearly  identify 
the  criterion  of  choice  while  conceding  that 
unquantifiable  policy  or  judgmental  aspects  affect  the 
choice  process.  Decision-makers  like  to  have  an 
"intuitive"  sense  that  the  decisions  they  make  are  valid. 
Therefore,  they  must  be  given  the  opportunity  to 
"assimilate  the  raw  data"  themselves.  This  is  especially 
critical  for  those  alternatives  that  are  characterized  as 
ones  that  "yield  more  but  cost  more."  Preference  in  this 
situation  is  a  very  judgmental  thing  and  circumstances 
and  risks  usually  affect  the  decision.  The  best 
approach,  therefore,  is  to  give  the  decision-maker  both 
effectiveness  and  cost  numbers  and  let  him  use  his  own 
judgement  guided  by  analytics. 

Where  Does  a  COEA  Fit  In  the  DoD  Management 
System 

The  new  acquisition  regulations  establish  a  DoD 
management  system  (Figure  4)  that  consists  of  three 
separate  but  supporting  decision  processes: 
Requirements,  Acquisition  and  PPBS. 


Each  process  supports  a  different  aspect  of  DoD 
activity  and  in  turn  each  is  supported  by  certain  types  of 
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analyses.  The  requirements  process,  for  instance,  is 
supported  by  Mission  Area  Assessments  (MAA)  and 
Mission  Needs  Analysis  (MNA).  An  MAA  identifies 
deficiencies  given  current  force  structures,  mission 

requirements  and  threats.  A  MNA  takes  these 

deficiencies  and  confirms  that  a  materiel  solution  is  the 
best  way  to  cover  them.  The  MNA,  therefore,  justifies 
an  acquisition  "new  start."  Requirements  evaluation  is 
the  responsibility  of  the  Joint  Requirements  and 
Oversight  Council. 

The  Acquisition  process  is  supported  directly 
by  Affordability  Analyses  and  by  COEAs  (referred  to 
as  COEA  1-4)  at  each  of  the  four  acquisition  milestones 
(Figure  5). 


Figure  5 


The  purpose  of  these  COEAs  is  discussed 
more  folly  below.  Acquisition  is  the  responsibility  of 
Under  Secretary  of  Defense  for  Acquisition  [USD(A)] 
and  is  overseen  and  managed  by  three  acquisition 
committees  and  by  a  Defense  Acquisition  Board. 

The  PPBS  does  not  require  pre-specified 
analyses.  However,  as  it  is  executed  to  evaluate  service 
programs,  embodied  in  the  Program  Objective 
Memorandum  (POM),  PPBS  usually  relies  on  a 
complex  of  studies,  issue  examinations,  quick  analyses, 
special-purpose  studies  and  COEAs.  PPBS  is  executed 
in  two  parts:  program  review  and  budget  review  with 
the  various  analyses  supporting  these  two  parts. 
Program  review  is  the  responsibility  of  the  Assistant 
Secretaiy  of  Defense  for  Program,  Analysis  and 
Evaluation  [ASD(PA&E)j  while  budget  review  is 
driven  by  the  Assistant  Secretary  Of  Defense 


(Comptroller)  (ASD(C)).  The  result  of  PPBS  is  the 
president's  budget. 

The  various  analyses  noted  above  support  and 

complement  each  other  as  a  way  of  interfacing  the  three 

DoD  decision  processes.  For  example  Requirements 
and  Acquisition  interface  at  milestone  zero  (MS0).  This 
is  the  "new  start"  milestone.  Thus  analyses  to  justify 
requirements  and  to  identify  deficiencies  occur  prior  to 
MS0  but  feed  into  the  MSO  review.  In  order  for  a 
system  to  begin  its  acquisition  path,  the  mission  need 
statement  must  be  supported  by  an  MAA  and  an  MNA. 
The  strength  of  the  MSO  decision,  therefore,  is  based  on 
the  qualify  of  the  MAA  and  the  MNA  done  during  pre- 
MS0  evaluations.  At  milestone  zero,  the  studies  and 
the  alternatives  for  COEA-1  are  established.  Clearly, 
the  scope  of  COEA-1  also  depends  on  the  level  of 
effort  reflected  in  the  MAA  and  MNA. 

The  interface  between  PPBS  and  Acquisition  is 
at  MS  1 .  This  is  a  key  milestone  in  that  the  decision  at 
this  point  is  a  "whole  enchilada"  decision.  The 
regulations  make  it  clear  that  proceeding  on  the  basis  of 
funding  wedges,  partial  programs,  or  promises  to  fund 
programs  at  a  later  time  will  not  be  acceptable.  Hence, 
newly-required  affordability  analysis  is  particularly 
important  because  it  makes  the  trade-off  in  a  given 
function  area  to  see  whether  proposed  programs  are 
affordable  within  total  fiscal  guidance.  The  COEA 
plays  a  relatively  minor  role  here  since  affordability  is 
strictly  a  cost  assessment.  But  the  COEA  can  support 
this  analysis  by  providing  effectiveness  measures  in  a 
given  mission  area  as  a  way  of  assessing  risks  in 
proceeding. 

The  COEA  supports  the  DoD  management 
system  in  different  ways.  It  is  best,  therefore,  to  view 
a  COEA  as  a  process  rather  than  as  a  product.  The 
COEA  begins  with  the  successful  passage  of  MSO  since 
it  is  one  of  the  studies,  arguably  the  primary  study, 
initiated  immediately  after  MSO.  The  alternatives  for 
this  COEA  are  defined  during  the  MSO  review.  The 
COEA  for  MSI  is  an  evaluation  of  alternative  concepts. 
That  is  to  say,  it  examines  the  different  ways  a  job  can 
be  done  or  a  mission  carried  out.  The  alternative  set  at 
this  stage  should  include  conceptually  different 
approaches  to  meet  the  requirement  and  should  not  be 
simply  a  variation  on  a  single  theme.  Preferably,  the 
COEA-1  should  be  done  at  the  strategic  or  operational 
level  based  on  a  broad  notion  of  mission 
accomplishment.  The  objective  of  this  COEA  is  to 
screen  out  those  approaches  that  do  not  "work"  and  to 
narrow  the  set  of  system  options. 
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COEA-2  should  focus  on  a  comparison  of 
systems  for  accomplishing  a  mission.  This  analysis 
should  deal  with  alternative  hardware  designs  or,  in  the 
case  of  non-developmental  items,  actual  systems. 
Prototypes  developed  during  Phase  1  and  performance 
estimates  from  them  should  be  used  as  the  basis  for 
estimating  the  combat  effectiveness  of  the  alternatives. 
The  best  level  of  evaluation  here  is  tactical  where  the 
particular  performance  features  of  a  system  can  be 
assessed.  Since  the  decision  to  be  made  at  this  point  is 
on  the  best  hardware  configuration,  the  highest  possible 
level  of  a  combined-aims  battle  that  still  draws  out 
hardware  differences  is  the  best  approach.  In  practice, 
this  frequently  turns  out  to  be  the  battalion  or  brigade 
level  for  Army  systems. 

COEA-3  occurs  on  an  exception  basis.  In 
particular,  it  is  done  only  if  substantial  changes  have 
developed  either  in  the  configuration  of  a  system,  its 
cost  or  the  threat.  Even  in  this  case,  however,  the 
question  to  be  answered  is  whether  the  change  is  drastic 
enough  to  invalidate  previous  cost-effectiveness 
rankings  and  preferences  not  whether  cost-effectiveness 
estimates  are  different.  This  COEA  is  done  at  the 
system  level  since  the  objective  is  to  certify  the  cost- 
effectiveness  established  by  COEA-2.  In  general, 
COEA-3  simply  updates  COEA-2  using  better 
performance  data  obtained  from  engineering  or  test 
results.  Unfortunately,  in  practice,  so  much  time  passes 
between  MSI  and  MS2  that  a  COEA-3  is  usually 
required.  While  this  may  seem  by  analysts  as  an 
unnecessary  burden,  from  a  management  viewpoint  it 
is  essential  because  the  decision  to  be  made  here  is 
usually  a  "big-bucks"  one,  namely,  should  the  system 
go  into  production.  At  this  stage,  there  is  rarely  little 
alternative  to  a  no-go  decision  except  terminating  a 
program.  Thus,  the  certification  of  cost-effectiveness  by 
COEA-3  is  a  critical  component  of  the  decision. 

COEA-4  is  also  optional.  It  closes  the  loop  on 
the  acquisition  cycle.  At  MS4  the  issue  is  whether  to 
upgrade  a  system  that  is  already  fielded  or  to  initiate  a 
new  system  acquisition.  Upgrade  options  are  the  main 
focus  in  the  alternative  set  for  this  COEA. 
Additionally,  replacement  systems,  existing  as  well  as 
new,  are  also  included  as  these  options  can  potentially 
drive  the  issue  back  to  the  requirements  phase  and  a 
subsequent  MSO.  The  context  for  this  COEA  is  mix  of 
assumptions  and  conditions  with  emphasis  ranging 
from  future  to  near-term  and  from  specific  systems  to 
general  concepts.  The  level  of  this  COEA  is  likewise 
a  mix  from  detailed  tactical  situations  to  large  scale 


operations.  The  real  utility  of  this  COEA  is  as  a  tool  to 
determine  whether  there  is  a  need  for  something  new  or 
whether  improvements  to  current  systems  will  suffice. 
COEA-4  serves  the  gut  modernization  issue  in  DoD. 

Generic  COEA  Methodology 

A  generic  methodology  for  doing  a  COEA  is 
shown  in  Figure  6.  The  key  inputs  are  at  the  top,  the 
key  processes  are  in  ovals  and  the  key  outputs  are  noted 
by  heavy  arrows. 


Inputs:  External  to  the  COEA  are  the 

alternatives,  the  operational  culture,  the  global  situation 
and  technology  projections.  Alternatives  are  defined 
either  through  outside  analyses  or  policy  decisions. 
The  COEA  begins  with  a  set  of  alternatives,  including 
a  basecase,  and  seeks  to  identify  the  one  preferred  from 
this  set.  Obviously,  unless  an  alternative  is  included  in 
the  input  set  it  will  not  be  recommended  at  the  end. 
Therefore,  a  crucial  consideration  at  the  beginning  of 
the  COEA  is  the  makeup  of  this  set.  It  should  be  as 
comprehensive  as  possible  but  allowances  should  also 
be  made  to  expand  it  later  as  additional  alternatives 
surface. 

Since  the  COEA  is  an  operational  evaluation, 
the  underlying  culture  of  the  "operators"  enters  in 
through  the  development  of  an  "operational  concept" 
for  the  study.  Simply  stated,  this  says  that  what  one 
believes  about  how  to  do  a  job  determines  what 
approaches  one  takes  to  doing  it.  For  example,  due  to 
different  operational  cultures,  the  operational  concept 
for  providing  point  air  defense  will  differ  among  the 
services.  While  the  Army  might  rely  on  guns  and 
missiles  on  mobile  platforms  at  relative  short  ranges. 
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the  Air  Force  might  emphasize  long-range  missiles  and 
manned  interceptors  and  the  Navy  mid-to-long  range 
air  and  missiles.  Thus,  a  key  outside  determinant  of  a 
COEA  is  the  accepted  operational  culture  reflected  in 
the  "way  to  fight”  as  trained  for  and  practiced  by  the 
services. 

The  COEA  must  be  meaningful  within  a 
realistic  context.  This  context  is  set  by  assumptions 
regarding  the  global  situation  and  leads  to  the 
specification  of  the  combat  scenarios  that  are  the  basis 
for  a  COEA.  Until  now,  the  global  context  was 
relatively  stable  and  rigid.  For  the  most  part  it  implied 
a  land  battle  scenario  in  Europe  emphasizing  the  heavy 
armor  battle.  Of  course,  this  is  now  changed.  New 
scenarios,  therefore,  must  be  developed  which 
accommodate  the  new  assumptions  about  the  global 
situation  and  which  can  be  used  to  drive  the  combat 
effectiveness  assessments  in  COEAs. 

Since  a  COEA  is  mainly  an  analysis  that 
supports  the  acquisition  process,  its  end  product  is  the 
identification  of  systems  that  can  potentially  be  added 
to  the  force  structure.  Consequently,  the  availability  of 
technology  becomes  an  important  aspect  of  defining 
alternative  systems.  However,  technology  projections 
are  outside  the  scope  of  the  COEAs.  They  are  external 
inputs  which  determine  not  only  alternatives  that  should 
play  but  also  the  level  of  performance  that  is 
achievable.  Realistic  projections  of  technology  are 
important  if  the  alternative  set  and  the  corresponding 
performance  parameters  are  to  be  credible.  A  highly 
optimistic  projection  will  cause  near  term  systems  to 
look  unduly  bad  while  a  pessimistic  one  will  preclude 
taking  advantage  of  real  opportunities. 

Processes:  In  its  most  general  form,  a  COEA 
is  a  collection  of  subanalyses  tied  together  in  such  a 
way  that  data  generated  by  one  is  used  by  another.  The 
first  level  of  processes  are  ones  that  look  at 
organization,  concept  of  operations,  scenario 
development,  and  performance  analysis.  These 
processes  translate  the  external  inputs  into  analytical 
formats  for  later  COEA  processes.  Organization 
Analysis  and  Operational  Concept  Development 
consider  each  of  the  alternatives  and  using  the 
operational  culture,  develop  an  organizational  structure 
that  will  function  militarily  to  carry  out  a  mission.  This 
activity  involves  defining  not  only  the  best  organization 
for  conducting  combat  operations  with  an  alternative 
but  also  the  quantities  of  and  support  required  by  each 
system.  The  product  of  these  processes  is  a  "combat 
force"  that  may  be  considered  a  microcosm  of  the  total 


force  that  would  exist  if  that  alternative  is  selected. 
This  combat  force  might,  and  probably  will,  be 
configured  differently  for  each  alternative.  It  is  not 
simply  the  case  where  the  same  strucuture  is  used  and 
alternative  systems  substituted.  Also,  the  combat  force 
will  include  support  systems  along  side  the  alternative 

systems  since  it  must  be  fully  mission  capable.  This 
force  is  sometimes  referred  to  as  a  "slice"  of  the  total 
force  structure. 

Scenario  analysis  generates  realistic  combat 
situations  to  use  as  study  scenarios.  It  generates  these 
at  various  levels  from  low-resolution,  theater  level 
operations  down  to  high-resolution,  small-scale  tactical 
situations.  The  level  depends  on  the  COEA  being  done. 
Regardless  of  level,  however,  the  study  scenarios  are 
defined  in  the  following  terms:  the  composition  of  the 
friendly  force  (i.e  the  combat  force),  the  opposing  force 
(i.e.  the  threat),  the  mission  to  be  accomplished  (i.e.  the 
military  objective),  the  timing  or  chronology  to  be 
followed  (i.e.  the  battle  dynamics)  and  the  environment 
in  which  the  operation  will  occur.  All  of  these 
components  must  be  specified  in  order  to  drive  the 
combat  models  used  later  in  the  methodology. 
Additionally,  the  complete  COEA  uses  more  than  one 
study  scenario.  The  spectrum  of  scenarios  should 
include  some  that  are  stressful,  non-stressful,  likely, 
worst  case  and  best  case.  At  all  costs,  however,  study 
scenarios  should  not  be  contrived  and,  if  possible,  they 
should  relate  back  to  the  Defense  Planning  Guidance. 
In  general,  a  broad  spectrum  of  scenarios  is  desirable 
since  the  point  of  the  analysis  is  to  determine  die  best 
alternative  under  a  variety  of  conditions. 

The  heart  of  the  COEA  is  the  Cost  Analysis  and 
the  Combat  Effectiveness  Analysis.  In  cost  analysis, 
comparative  LCC  estimates  are  developed  for  each 
alternative.  The  two  critical  aspects  of  this  analysis  are 
realism  and  completeness.  Realism  is  attained  by  using 
the  latest,  most  accurate  data  about  each  of  the 
alternative.  It  is  achieved  by  having  cost  analysts  work 
closely  with  and,  ideally,  even  live  with  system 
designers  and  developers.  Also,  since  cost  analysts 
usually  work  either  parametrically,  bottoms-up  or  by 
analogy,  extensive  (in  terms  of  scope  and  history)  data 
bases  are  essential.  The  operative  concept  is  that  a  cost 
analyst  without  a  good  data  base  is  a  cost  analyst  not  to 
be  believed.  Completeness  is  attained  by  including  all 
factors  that  contribute  to  the  cost  of  an  alternative.  This 
means  that  the  life  cycle  cost  estimate  should  include 
more  than  just  the  "big  five"  cost  components  of 
procurement,  R&D,  Military  Construction,  Personnel 
and  Operations  &  Support.  Other  costs,  such  as  sunk 
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and  phase-in  costs,  should  also  be  included.  An 
estimate  with  costs  hidden  or  omitted  will  lead  to  an 
erroneous  basis  for  comparison.  The  COEA  will  only 
be  as  good  as  the  "comparative  "  life  cycle  cost 
estimates  of  each  alternative. 

Combat  effectiveness  analysis  is  a  measure  of 
an  alternative's  ability  to  meet  established  mission 
requirements  in  an  operational  environment.  It  is  to  be 
distinguished  from  performance  analysis  which  is  a 
measure  of  how  well  a  specific  system  accomplishes  an 
assigned  task.  Operational  effectiveness  is  a  force 
attribute;  performance  is  a  system  attribute.  In  a 
COEA,  combat  effectiveness  is  quantified  through 
"force  level"  measures  of  effectiveness  (MOE).  The 
ideal  MOE  will  measure  mission  accomplishment. 
However,  since  in  many  cases  this  is  hard  to  quantify, 
lower  order  MOE  are  used  as  surrogates  and 
supplemented  with  judgment.  Often  these  surrogates 
focus  on  casualties,  both  enemy  and  friendly,  and  are 
quantified  as  ratios.  Typical  among  them  are  Loss 
Exchange  Ratios  (LERs)  and  Force  Exchange  Ratios 
(FERs).  Combat  effectiveness  is  usually  quantified  via 
simulation  or  tests.  Simulations,  whether  of  the  Man- 
In- The-Loop  or  closed-form  variety,  seek  to  measure 
combat  outcome.  They  play  through  a  battle  as  defined 
by  the  scenario,  and  evaluate  battle  outcome  in  terms  of 
casualties  or  objectives  achieved,  based  on 
performance,  tactics,  and  environment.  At  the  tactical 
level,  these  models  tend  to  be  complex  stochastic 
simulations  used  as  the  vehicle  for  executing  an 
experiment  design.  Numerous  replications  are  made  to 
achieve  statistical  stability  and  the  model  results  are 
reported  at  given  confidence  levels.  At  the  operational 
or  higher  level  where  representing  individual  platforms 
is  not  critical,  simulations  are  normally  of  the 
deterministic  type  and  not  infrequently  lanchester- 
equation  based.  In  these  models,  the  critical  part  of  the 
analysis  is  obtaining  estimates  of  attrition  coefficients. 
Once  coefficients  are  known,  however,  results  are 
quantified  in  one  pass,  i.e.  one  replication  of  the  model. 
Confidence  levels  for  model  results  in  this  case  are 
determined  primarily  by  the  confidence  level  of  the 
inputs.  In  either  type  of  model,  high  or  low  resolution, 
the  key  is  to  play  an  integrated  battlefield.  Meaningful 
combat  effectiveness  analysis  requires  a  combined- 
arms  battle  situation  so  that  the  value  of  a  system  is 
determined  while  taking  into  account  the  synergistic 
effects  of  all  combat  force  elements. 

Since  combat  effectiveness  analysis  is 
essentially  a  computer  experiment,  analysis  of  model 
output  should  be  done  using  sound  statistical  principles. 


In  essence,  the  study  is  an  experiment  and  one  model 
replication  is  an  observation.  The  analyst's  job  is  to 
define  the  experimental  design  to  answer  specified 
questions  (called  the  essential  elements  of  analysis 
(ERA))  The  design,  usually  defined  as  a  "runs  matrix," 
must  account  for  the  limitations  of  the  model  and  must 
also  provide  for  sufficient  collection  of  data  to  meet 
given  confidence  levels.  Model  execution  should  not 
be  strictly  a  number  crunching  exercise  as  the  dynamics 
of  the  battle  must  reflect  both  experience  and  the  real 
world  if  the  results  are  to  be  believed.  Operational 
effectiveness  analysis  is  partly  an  art  form  but  it  is  one 
that  any  serious  analyst  can  leam  and  use  meaningfully 
in  decision-making. 

The  integration  of  cost  and  effectiveness  results 
is  the  most  difficult  of  the  COEA  processes.  This 
problem  is  at  once  analytical  and  political. 
Sophisticated  techniques,  ranging  from  simple  ratios  to 
complex  multi-attribute  techniques  to  the  more  recent 
analytic  hierarchy  process,  can  be  applied  to  identify 
alternative  preferences  .  However,  the  more 
complicated  these  techniques  are,  the  less  visibility  the 
decision-maker  has  into  them  and  the  less  he  will  rely 
on  them  to  make  his  choices.  This  suggest  that  the 
analyst  should  strive  to  implement  the  tools  but  should 
also  make  the  data  available  to  the  decision-maker  so 
he  can  make  his  own  assessments. 

A  discussion  of  the  methodology  of  a  COEA  is 
not  complete  without  a  few  comments  on  judgment. 
There  are  several  kinds  of  judgment  that  enter  into 
completing  a  COEA:  one  is  the  judgment  that  comes 
from  being  an  experienced  analyst,  another  from  being 
experienced  in  military  operations  and  a  third  from 
being  politically  wise.  The  first  kind  of  judgment 
allows  one  to  know  when  analysis  tools  are  functioning 
as  intended.  It  is  not  an  uncommon  COEA  failing  for 
models  to  run  (i.e  execute)  but  to  do  so  without 
producing  useful  output.  Analysis  judgment  is  the  only 
thing  that  reveals  this  problem.  It  is  also  analytical 
judgment  that  allows  one  to  suspect  that  input  data  is 
wrong.  The  inexperienced  analyst  will  usually  carry 
on,  not  even  knowing  there  is  a  problem;  the 
experienced  analyst  will  stop  and  check.  Military 
judgment  is  equally  important  in  COEAs.  Through  it 
comes  recognition  of  the  "worth"  of  a  mission;  i.e 
whether  the  job  is  worth  doing  in  order  to  accomplish 
the  military  objective.  Only  one  with  military 
experience  can  judge  whether  a  mission  is  a  reasonable 
one.  Military  judgment  also  enters  into  knowing  when 
or  to  what  degree  a  mission  has  been  accomplished. 
Trite  as  it  may  sound,  only  the  experienced  combat 
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analyst  can  truly  judge  when  a  battle  is  over.  Finally 
there  is  "political"  judgment  which  comes  to  bear  when 
the  results  of  COEAs  are  not  clear  cut.  In  the  really 
interesting  decision  problems,  alternatives  that  are 
decisively  good  or  bad  are  the  exception.  The  typical 
defense  "choice  problem"  is  one  where  there  is  gain  but 

there  is  cost.  In  this  case  the  decision-maker  is 
expected  to  exercise  his  intuition,  take  the  risk  if 
appropriate,  and  move  the  process  forward  as 
efficiently  as  possible.  No  analysis  is  ever  going  to 
replace  judgments.  A  truly  good  analysis,  therefore, 
will  provide  information  to  make  these  judgments 
sounder.  That  is  the  true  value  of  a  COEA. 

How  Is  A  COEA  Evaluated  For  Quality? 

There  are  certain  elements  and  criteria  that  can 
be  used  to  make  quality  judgments  about  COEAs. 
While  it  is  unreasonable  to  develop  a  checklist  for 
quality,  it  is  possible  to  identify  characteristics  which 
make  a  good  COEA.  The  following  list  serves  this 
purpose.  It  poses  questions  to  check  whether  a  COEA 
has  omitted  or  abused  some  aspect  of  the  analysis 
requirement  or  procedure.  Note  that  there  are  no  right 
or  wrong  answers.  The  questions  serve  only  to  guide 
the  judgment  of  whether  the  COEA  has  covered  all  the 
elements  and  if  so,  whether  it  has  done  so 
comprehensively.  Note  also  that  the  assumption  for 
these  guidelines  is  that  a  MSO  has  been  successfully 
passed  since  the  first  time  a  COEA  is  required  is  after 
this  has  happened. 

The  guidelines  are  given  in  terms  of  various 
components  or  "elements"  of  a  COEA  (Table  1). 


COEA  ANALYSIS  ELEMENTS 


IN  THE  5000  REGS 

Mission  Needs  Analysis 
Threat 

Operational  Environments 

Constraints  &  Assumptions 

Operational  Concepts 

Functional  Objectives 

Alternatives 

Models 

Data 

Measures  of  Effectiveness 
Costs 

Trade-off  Analyses 
Analysis  of  Alternatives 
Conclusions 


NOT  IN  THE  5000  REGS 

Issues  &  Essential 
Elements  of  Analysis 
Study  Plan 
Related/Sub-analyses 
Scenarios 
Methodology 


Table  I 


This  partition  is  a  matter  of  convenience  but  it  serves 
the  purpose  of  organizing  a  quality  check  on  the 
analysis.  Also,  for  reference,  the  elements  are 
identified  as  either  being  explicitly  addressed  in  the 


acquisition  regulation  or  not.  This  carries  no 
connotation  of  their  relative  importance  and  is  noted 
only  for  reference. 


Element:  Mission  Needs  Analysis 

Guidelines: 

1.  Is  the  MSO  justification  still  valid?  In 
particular, 

a.  are  the  study  scenarios  still  suitable 
ones  for  the  operational  analysis? 

b.  are  the  weapon  systems  in  the 
combat  mini-force  still  the  same 
ones  or  is  there  a  difference  in  the 
synergistic  effects  represented  by 
the  analysis? 

c.  is  the  time  frame  of  the  analysis 
still  the  same  or  has  there  been  a 
change  in  schedules  for  the 
availability  of  equipment  on  either 
side? 

2.  Is  a  new  level  of  analysis  (i.e  degree  of 
resolution)  required?  In  particular 

a.  has  there  been  any  change  in  the 
basis  of  issue  for  the  item  under 
investigation? 

b.  have  new  tactics  or  doctrine  been 
introduced  to  change  the  "way  to 
fight"  assumptions? 

Element:  Threat 

Guidelines: 

1.  Is  the  assumed  threat  consistent  with 
intelligence  assessment?  In  particular, 

a.  has  the  threat  been  validated  by  the 
Defense  Intelligence  Agency? 

b.  is  the  validation  recent  enough, 
relative  to  known  world  changes, 
to  accept  it  at  face  value? 

2.  Is  the  threat  defined  in  a  way  that  is  consistent 
with  realistic  operational  constraints?  In 
particular, 

a.  is  the  threat  reasonable  for  the 
time,  place  and  type  of  operation 
so  that  it  is  neither  10  feet  tall  or  a 
pushover? 

b.  has  a  reasonable  level  of 
intelligence  been  given  to  the 
threat  force  so  that  it  is  neither  too 
dumb  nor  too  all-knowing? 
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Element:  Operational  Environment 

Guidelines: 

1.  Have  contributions  from  supporting  forces, 
either  from  sister  services  or  allies,  been 
accounted  for  so  that  the  proper  integrated 
perspective  is  reflected? 

2.  Has  the  right  timeframe  (i.e.  season)  of  the 
operation  been  setup? 

3.  Have  realistic  assumptions  concerning  the 
availability  of  support  or  resources  (personnel 
and  funding)  been  made? 

4.  Are  the  assumptions  about  the  availability, 
quantity,  distribution  and  readiness  of  the 
equipment  item  realistic? 

Element:  Study  Constraints  and  Assumptions 

Guidelines: 

1.  Are  the  alternatives  appropriate  for  the  level 
of  analysis:  concepts  at  MSI,  designs  at  MS2, 
systems  at  MS3  and  upgrade  options  at  MS4? 

2.  Is  the  alternative  set  broad  enough  or  is  it 
simply  a  list  of  narrow  variations  on  the  same 
theme? 

3.  Have  reasonable  alternatives  been  eliminated 
simply  by  assumption  about  environment  or 
performance? 

4.  Are  projections  dealing  with  the  availability, 
reliability,  capability,  etc  of  force,  equipment, 
technology  and  funds  realistic? 

Element:  Operational  Concept 

Guidelines: 

1.  Is  the  employment  doctrine  (i.e  the  how  to 
fight  assumptions)  feasible  and  valid  vis-a-vis 
training? 

2.  Does  the  assumed  force  structure  reflect 
realistic  plans  or  proposals  for  the  overall 
composition  of  the  US  force? 

3.  Have  alternative  employment  doctrines  been 
considered  to  ensure  effectiveness  does  not 
suffer  from  poor  utilization? 

4.  Has  field  or  test  experience  been  folded  into 
the  analysis  so  that  untried  theoretical 
procedures  are  not  included? 

Element:  Functional  Objectives 

Guidelines: 

1.  Has  a  task  list  been  specified  for  each 
alternative  that  reflects  the  operational  concept 
for  it? 

2.  Have  tasks  been  defined  in  such  as  way  as  to 
permit  the  quantification  of  accomplishment 
or  completion? 


3.  Have  key  performance  parameters  for 
alternative  systems  been  captured  by  the  tasks 
defined  for  the  operation? 

4.  Are  the  measures  of  system  performance 
measurable  and  testable? 

Element:  Alternatives 
Guidelines: 

1 .  Has  the  study  basecase  been  identified? 

2.  Does  the  alternative  set  cover  a  good  spectrum 
of  choices?  In  particular, 

a.  are  current  systems  included? 

Have  they  been  "undersold"? 

b.  are  product  improvements  to 

current  systems  included? 

c.  are  prospective  near-term  and  far- 
term  systems  included? 

d.  are  conceptual  or  "paper"  systems 

included?  Have  they  been 
"oversold"? 

3.  Have  alternatives  that  vary  only  tactics  or 
doctrine  been  considered? 

4.  Have  alternatives  that  represent  "a  different  way 
to  do  the  job"  been  included? 

Element:  Models 
Guidelines: 

1 .  Do  the  models  applied  in  the  study  quantify  the 
defined  measures  of  effectiveness  (MOE), 
performance  (MOP)  and  costs? 

2.  Has  the  result  of  the  combat  effectiveness  been 
done  on  a  force-on-force  (i.e  an  integrated 
combined-arms)  basis? 

3 .  If  the  study  model  is  a  man-in-the-loop  type,  has 
the  human  "bias"  been  eliminated  or  identified 
from  the  results? 

4.  If  the  study  model  is  extant,  does  it  apply 
directly  to  the  problem  or  has  it  been  properly 
modified  to  fit? 

5.  If  the  study  model  is  new,  has  it  been  certified 
as  representative  of  the  scenario  and  systems 
under  investigation? 

6.  Are  the  model  process  transparent  and 
explainable  at  reasonable  levels  of  analytical 
sophistication? 

7.  Has  the  study  model  been  "tested”  by 

a.  running  a  known  system  (often  the 

basecase)  and  checking  for 

"reasonableness"? 

b.  running  a  case  that  uses  current 

friendly  systems  opposed  by 
current  enemy  systems  and 

comparing  with  other  "net  assessments"? 
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Element:  DATA 

Guideline: 

1 .  Are  the  study  data  current? 

2.  Are  the  study  data  accurate  or  is  there  a 

known  level  of  precision  for  it? 

3.  Are  the  data  technically  sound,  derived  by 
rigorous  engineering  analyses  or  are  they 
"anecdotal"? 

4.  Have  the  data  been  verified  by  (or  collected 
from)  relevant  developmental  or  operational 

.  ■  ? 

5.  Do  the  data  accurately  reflect  assumptions 
about  tactics  and  doctrine? 

Element:  Measures  of  Effectiveness  (MOE) 

Guidelines: 

1.  Are  the  study  MOE  quantifiable,  directly  or 
indirectly  from  the  model  outputs? 

2.  Do  the  MOE  discriminate  among  the 
alternatives? 

3 .  Are  the  MOE  force-level  measures? 

4.  Do  the  MOE  measure  battle  outcome  or 
mission  accomplishment? 

5.  If  the  MOE  are  "aggregated"  from  other 
measures  using  weights  or  ratios,  do  they 
distort  the  relative  comparisons? 

Element:  Cost 

Guidelines: 

1 .  Is  the  cost  estimating  technique  used  valid  for 
the  type  and  level  of  analysis? 

2.  Are  the  quantities  and  schedules  used  to  make 
the  estimates  valid? 

3.  Are  the  input  data  used  to  make  the  cost 
estimates  current? 

4  Is  the  input  data  valid?  In  particular, 

a.  do  they  conform  with  the 
operational  concept? 

b.  have  hey  been  properly  collected? 

5.  Has  a  comparative  life  cycle  cost  estimate 
been  developed  for  each  alternative  so  that 
comparison  is  on  the  same  basis? 

6.  Are  the  cost  estimates  consistent  with  other 
estimates,  especially,  the  baseline  cost 
estimate  (BCE)  and  the  independent  cost 
estimate  (ICE)? 

Element:  Trade-off  analyses  (sensitivity  and 

uncertainty) 

Guideline: 

1.  Has  the  study  considered  "analysis 
excursions"  in  the  following  areas:  a)  threat 
capabilities,  b)  friendly  capabilities,  c) 


environmental  conditions,  d)  countermeasures, 
e)  availability  or  level  of  support  (either  combat 
service  support  or  complementary  combat 
systems),  f)  availability  or  level  of  funding? 

Element:  Analysis  of  alternatives 

Guidelines: 

1 .  Has  a  comparison  been  given  on  an  equal-cost 
or  equal-effectiveness  basis  and,  if  so,  does  it 
reflect  alternatives  on  bases  consistent  with  the 
operational  concept  and  scenario? 

2.  Have  absolute  values  of  the  MOEs  been  given? 

3.  If  a  composite  (i.e.  weighted)  MOE  is  used,  are 
the  weights  explained  in  a  meaningful  sense? 

4.  If  ratios  are  used,  are  they  unambiguous  in 
meaning? 

5.  Have  all  MOE  been  shown? 

6.  If  some  form  of  multi-attribute  or  decision 
analysis  technique  is  used,  have  the  inputs  to  it 
been  given  to  the  decision-maker  also? 

7.  Have  the  alternatives  been  ranked  and  a 
preference  indicated? 

8.  Have  the  alternatives  been  grouped  by  cost?  by 
effectiveness? 

9.  Have  the  alternatives  been  further  examined  for 
conditions  that  would  weaken  or  overturn  its 
preference  order? 

Element:  Conclusions 

Guidelines: 

1 .  Are  the  study  conclusions  exclusively  supported 
by  study  results  or  are  study  results  being 
complemented  by  outside  information  or 
knowledge? 

2.  Are  the  study  conclusions  responsive  to  the 
study  issues? 

Element:  Issues  or  Essential  Elements  Of  Analysis 

(EEA) 

Guidelines: 

1 .  Is  the  study  responding  to  the  current  decision 
issue  or  has  time  rendered  it  irrelevant  or 
peripheral? 

2.  Does  the  study  respond  to  relevant  "Acquisition 
Decision  Memorandum"  of  the  regulations? 

3.  Do  the  essential  elements  of  analysis  of  the 
study  properly  reflect  the  decision  issues? 

4.  Are  the  EEA  comprehensive  vis-a-vis  the 
issues? 

Element:  Study  Plan 

Guidelines: 
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1 .  Has  the  plan  been  coordinated  at  all  levels  of 
review  up  to  and  including  OSD? 

2.  Has  the  plan  been  properly  resourced  in  terms 
of  time,  people,  and  money? 

Element:  Scenarios 
Guidelines: 

1 .  Does  the  study  include  more  than  one 
scenario? 

2.  If  several  scenarios  are  used,  are  they  a  good 
spectrum?  In  particular,  do  they  include 

a.  a  stressful  one? 

b.  a  light  one 

c.  a  likely  one? 

d.  a  wcrst/best  case? 

3.  Is  the  scenario  "contrived"  to  show  an 
alternative  as  favorable? 

4.  Can  the  scenario  be  traced  back  to  the  Defense 
Planning  Guidance? 

5.  If  the  scenario  is  not  traceable  to  the  DPG,  has 
it  been  validated  by  the  appropriate 
"warfighters"? 

Element:  Methodology 
Guidelines: 

1.  Does  the  COEA  methodology  answer  all  the 
Study  EEA? 

2.  Does  the  methodology  include  enough 
flexibility  to  allow  mid-course  corrections? 
Does  it  identify  decision  "points  of  no  return"? 

3.  Does  the  methodology  allow  for  periodic 
progress  reports  and  especially  a  final  report? 


Element:  Related/Sub-analyses 
Guidelines: 

1 .  Have  all  studies  supporting  the  same  acquisition 
action  as  the  COEA  been  synchronized  with 
respect  to  timeframe,  operational  concept, 
scenario,  technology,  performance  data,  etc. 

2.  Has  the  linkage  to  operational  tests, 
development  of  requirement  documents  and 
program  baseline  been  made  explicit? 

Summary 

COEAs  in  support  of  the  acquisition  process  are 
an  established  way  of  doing  business  in  DoD. 
Decision-makers  expect  sound  analyses  at  each 
milestone  to  assist  them  in  making  the  tough  choices. 
COEAs  clearly  fulfill  this  requirement  now.  In  the 
future,  perhaps  modified  to  respond  to  the  changing 
security  environment  and  issues,  they  will  continue  to 
play  a  significant  role  in  DoD  analysis  efforts. 
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FUTURE  DIRECTIONS 
ARMY 

FOUR  CHALLENGES: 

•  DETERMINING  ARMY  MISSION  NEEDS  IN  A  JOINT 

OPERATIONAL  CONTEXT  THAT  ADEQUATELY 

REPRESENTS  THE  CAPABILITIES  OF  OTHER 
SERVICES. 

•  DEVELOPING  A  CLEARER  DEFINITION  OF  REGIONAL 
THREATS  AND  MILITARY  CAPABILITIES  REQUIRED 
TO  MEET  THE  GOALS  ESTABLISHED  BY  THE 
NATIONAL  MILITARY  STRATEGY. 

•  DEVELOPING  A  CLEARER  DEFINTION  OF  DECISIVE 
FORCE  AS  A  BASIS  FOR  ESTABLISHING  REQUIRED 
CAPABILITIES. 

•  DEVELOPING  A  CLEARER  PICTURE  OF  OUR 
ACQUISITION  STRATEGY,  ESPECIALLY  THE 
POTENTIAL  FOR  MOVING  FEWER  SYSTEMS  FROM 
THE  R&D  STAGE  TO  FULL  SCALE,  HIGH  RATE 
PRODUCTION. 

PRESENTED  BY  MR.  HOLLIS 


FUTURE  SITUATION 
AIR  FORCE 


EXPECT  DECREASING  RESOURCES 

EXPECT  INCREASED  SCRUTINY  AND  OVERSIGHT 

FOR  FUTURE  STARTS  (MILESTONE  1)  AND  OTHER 
ACQUISITION  MILESTONES,  WE  WILL  NEED: 

•  STRONGER  AND  MORE  COHERENT  RATIONAL 

•  BETTER  SUPPORTING  ANALYSIS 


PRESENTED  BY  MAJ  GEN  J.W  RALSTON 
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OVERALL  PROCESS 


MISSION  NEED  STATEMENT 


•  THE  MAJCOM  WRITES  AND  ISSUES  A  MISSION 

NEED  STATEMENT 

•  THE  MISSION  NEED  STATEMENT  STATES  THAT  A 
PARTICULAR  OPERATIONAL  OBJECTIVE  NEEDS 
INCREASED  EMPHASIS  AND  WHY 

•  THE  MNS  IS  VALIDATED  BY  THE  JOINT 
REQUIREMENTS  OVERSIGHT  COUNCIL  (JROC) 


MILESTONE  0 
(AN  EVENT) 


DIRECTION  TO  COMMENCE 
CONCEPT  EXPLORATION 


96 


USING  MAJCOM  RESPONSIBLE 
FOR  THE  PROCESS 


•  THE  MAJCOM  -  SUPPORTED  BY  DEVELOPERS  - 
FORMULATES,  DEFINES  AND  EVALUATES  VARIOUS 
OPERATIONAL  CONCEPTS  TO  ENHANCE  OUR 
ABILITY  TO  ACHIEVE  THE  STATED  OPERATIONAL 
OBJECTIVE 

•  THE  PROCESS  IS  SUPPORTED  BY  A  “CONCEPT 
EXPLORATION  ANALYSIS"  (CEA) 


ENHANCED  THROUGH  USE  OF  CAMPAIGN,  ENGAGEMENT 
AND  ENGINEERING  LEVEL  MODELS  AND  ANALYSIS 


COST  AND  OPERATIONAL 
EFFECTIVENESS  ANALYSIS 
(COEA) 


THE  MAJCOM  BASED  ON  THE  ANALYSIS  IS  RESPONSIBLE 
FOR  A  COEA  WHICH  DESCRIBES 

•  THE  RELEVANCE  OF  ACHIEVING  OPERATIONAL 
OBJECTIVES  (AND  ACCOMPLISHING  ATTENDANT  TASKS) 

•  THE  OPERATIONAL  CONCEPTS  SELECTED  AND  WHY 

•  THE  COST  EFFECTIVENESS  OF  EACH  SYSTEM 
CONSIDERED  TO  IMPLEMENT  THE  RELEVANT 
OPERATIONAL  CONCEPT 

•  THE  SELECTED  SYSTEM  AND  WHY  IT  WAS  CHOSEN 


ENHANCED  THROUGH  USE  OF  CAMPAIGN,  ENGAGEMENT 
AND  ENGINEERING  LEVEL  MODELS  AND  ANALYSIS 
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COEA  PROPOSAL  TO 
INITIATE  ACTION 


FOR  ACAT  I  PROGRAMS,  PREPAREED  BY  ASN 
(RD&A  DASN)  IN  COORDINATION  WITH  PROGRAM 
SPONSOR,  PROGRAMMANAGER  AND  THE  APPROPRIATE 
SYSCOM/PEO 

FOR  ACAT  II  &  III  PROGRAMS,  PREPARED  BY  PROGRAM 
SPONSOR  IN  COORDINATION  WITH  PROGRAM  MANAGER 
AND/OR  APPROPRIATE  SYSCOM/PEO 

CONTAINS  RECOMMENDATIONS  FOR 

-  CO-CHAIRS  OF  COEA  OVERSIGHT  BOARD 

-  ORGANIZATIONS  REPRESENTED  ON  THE  OVERSIGHT 
BOARD 

-  STUDY  DIRECTOR 

-  ORGANIZATIONS  TO  PROVIDE  STUDY  TEAM  MEMBERS 

-  SCHEDULE 

APPROVED  BY  OP-08  AND  ASN  (RD&A) 


STUDY  DIRECTOR  AND  STUDY  TEAM 


STUDY  DIRECTOR: 

•  PLANS  AND  SUPERVISES  COEA  STUDY 

•  COORDINATES  FUNDING  THROUGH  PROGRAM  MANAGER 

•  ESTABLISHED  STUDY  TEAM  MEMBERSHIP 
•MUST  HAVE 

-  STRONG  ANALYSIS  BACKROUND 

-  TECHNICAL  AND  OPERATIONAL  CREDIBILITY 

-  INDEPENDENCE  FROM  PROGRAM  MANAGER 


STUDY  TEAM  MEMBERS  DRAWN  FROM: 

-  APPROPRIATE  DASN  -  PROGRAM  OFFICE 

-  OPNAV  SPONSOR  -  OP-08,  OP-07,  OP-091 ,  OP, 092 

-  SUPPORTING  SYSCOM  -  FIELD  ACTIVITIES  AS  RQD 

-  PEO/DRPM 


DIRECTOR  &  TEAM  MEMBERS  MUST  BE  CHOSEN  TO  AVOID 
CONFLICT  OF  INTEREST  IN  FOLLOW-ON  EFFORTS 
RESULTING  FROM  ANALYSIS 
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FUTURE  DIRECTIONS 
USMC 


MOSTLY  SMALL  PROGRAMS 
ACAT  III  &  IV 
AAA  IS  ONLY  ACAT  I 

USMC  FOLLOWING  DON  POLICY 

OVERSIGHT  BOARDS 
STUDY  DIRECTORS 

EMPHASIS  ON  USER-ACQUISITION  TEAMWORK 

PROCESS  IS  TAILORED  TO  PROGRAM  SIZE 

HEADED  BY  HQ  USMC  R&P  AND  DEPUTY  EFP 
IN-HOUSE  SUPPORT  FROM  MCCDC, 
MARCORPSYSCOM  AND  OTHER  HQ  ELEMENTS 
FFRDC  AND  OTHER  OUTSIDE  SUPPORT 


PRESENTED  BY  MR.  BELEN 
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4.1  Working  Group  Charter 
Working  Group  1:  Test  &  Evaluation 


CHAPTER  4 

WORKING  GROUP  REPORTS 


Objective:  Define  and  report  on  the  role  of 

operations  research  in  the  COEA  process;  issues, 
problems,  and  research  areas,  and  the  future  directions 
for  military  operations  research  analysis  to  support 
COEAs  as  it  relates  to  this  working  group. 

Suggested  Issues: 

•  What  analytic  techniques  can  be  developed  to: 

•  Help  COEA  MOEs,  scenarios,  and  TEMPS 
be  consistent? 

•  Relate  "real  world"  data  to  COEA  model 
validation? 

•  Relate  developmental  test  data  to  COEAs? 

•  Relate  operational  test  data  to  COEAs? 

•  Relate  live  fire  test  data  to  COEAs? 

•  What  simulation  techniques  can  be  used  to  support 
tests  and  COEAs? 

Working  Group  2:  Threat  Assessments  and 

Scenarios 

Objective:  Define  and  report  on  the  role  of 

operations  research  in  the  COEA  process;  issues, 
problems,  and  research  areas,  and  the  future  directions 
for  military  operations  research  analysis  to  support 
COEAs  as  it  relates  to  this  working  group. 

Suggested  Issues: 

•  What  is  the  relationship  of  threat  assessments  and 
scenarios,  the  COEA  process,  and  operations  research? 

•  What  problem  areas  are  evident  in  the  above 
relationship? 

•  How  can  a  COEA  be  executed  as  the  "threat" 
evolves  during  the  timeframe  of  the  COEA? 


•  What  is  the  best  way  to  analytically  represent  a 
regional/contingency  threat  scenario? 

•  What  spectrum  of  scenarios  brings  out  the  versatility 
of  a  system  for  comparison?(High,  Mid,  Low)(Day, 
Night)  (High,  Hot)  (Low,  Cold) 

•  How  can  scenario  development  be  integrated  into 
the  COEA  process? 

•  How  do  you  set  up  study  scenarios  to  address  the 
spectrum  of  conflict  in  models  and  simulation? 

•  How  do  scenarios  cover  the  scope  to  accommodate 
all  phases  of  operations  from  pre-deployment  on? 

•  How  do  you  avoid  stacking  the  deck  through 
scenario  selection? 

•  What  is  the  appropriate  level  of  scenario  detail  at 
each  milestone? 

Working  Group  3:  Effectiveness  Methodologies 

Objective:  Define  and  report  on  the  role  of 

operations  research  in  the  COEA  process;  issues, 
problems,  and  research  areas,  and  the  future  directions 
for  military  operations  research  analysis  to  support 
COEAs  as  it  relates  to  this  working  group. 

Suggested  Issues: 

•  Models/Data 

•  What  are  the  spectrum  of  ways  to  simulate 
warfare/battles? 

•  What  are  the  characteristics  of  combat/cost 
models/simulations  to  conduct  the  COEA? 

•  How  is  models/data  credibility  achievable? 

•  What  are  the  unique  modeling  requirements 
to  address  campaign  analyses? 

•  What  is  the  value  of  a  DoD  clearing  house 
for  models/data? 
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•  How  can  Operational  Utility  Evaluations  relate  to 
COEAs? 

•  How  do  we  integrate  cost  parameters  and 
effectiveness  measures? 

•  How  can  effectiveness  methodologies  evolve  to 
accommodate  the  increased  emphasis  in  deployment, 
sustainment,  and  campaigns? 

•  How  do  we  model  the  evolving  high  technology 
battlefield? 

•  How  can  sensitivity  analyses  be  incorporated  into 
the  COEA? 

•  What  tools  are  available  to  accomplish  the 
sensitivity  analysis? 

•  Measures  of  Effectiveness 

•  How  do  we  show  or  measure  the  impact  of 
non-attrition  systems:  Command  &  Control, 
Reconnaissance,  Intelligence  Collection,  etc. 

•  Value  added  Methodologies? 

•  Statistical  comparisons 

•  Design  of  experiments 

•  Others? 


Working  Group  4:  Costing  Methodologies 

Objective:  Define  and  report  on  the  role  of 

operations  research  in  the  COEA  process;  issues, 
problems,  and  research  areas,  and  the  future  directions 
for  military  operations  research  analysis  to  support 
COEAs  as  it  relates  to  this  working  group. 

Suggested  Issues: 

•  How  can  it  be  ensured  the  costs  for  the  preferred 
option  on  the  COEA  match  the  costs  briefed  to  the  Cost 
Analysis  Improvement  Group? 

•  Affordability  --  What  is  the  linkage  between 
affordability  considerations  in  COEAs  and  in  POMs? 

•  Affordability  —  How  should  affordability  be 
addressed  at  Milestone  reviews? 


•  What  techniques  are  available  to  perform 
affordability  analysis? 

•  How  do  we  integrate  cost  parameters  and 
effectiveness  measures? 

•  What  is  the  best  way  to  provide  cost  analysis  support 
to  the  decision  maker? 

•  What  role  should  life  cycle  costing,  decision  costing, 
and  present  value  analysis  play  in  COEAs? 

•  How  do  we  ensure  consistency  between  the  COEA, 
Independent  Cost  Estimate  (ICE),  Independent  Cost 
Analyses  (ICA),  and  Baseline  Cost  Estimate(BCE)? 

•  How  can  sensitivity  analyses  be  incorporated  into 
the  COEA? 

•  What  tools  are  available  to  accomplish  the 
sensitivity  analysis? 

•  Are  current  costing  methodologies  adequate  to 
compare  the  alternatives  in  a  COEA? 

•  What  costing  methodologies  should  be  developed  to 
advance  the  state  of  the  art  in  COEA? 


Working  Group  5:  COEA  Management 

Objective:  Define  and  report  on  the  role  of 

operations  research  in  the  COEA  process;  issues, 
problems,  and  research  areas,  and  the  future  directions 
for  military  operations  research  analysis  to  support 
COEAs  as  it  relates  to  this  working  group. 

Suggested  Issues: 

•  What  are  the  roles  of  the  service.  Program  Manager, 
contractor,  and  Federally  Funded  Research  and 
Development  Centers  (FFRDCs)  in  COEAs? 

•  How  do  you  structure  a  Study  Advisory  Group 
(SAG)  to  support  the  COEA  process? 

•  What  should  be  the  role  of  SAGs  in  COEA 
development? 

•  Should  SAGs  be  monolithic  (one  SAG  with  all 
players)  or  tiered  (several  SAGs  at  Service,  OSD,  etc.)? 
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Models  (Effectiveness/Cost) 


•  How  do  we  coordinate  the  efforts  of  the  contributors 
including  threat,  data,  modelling,  cost,  tests,  and 
others? 

•  How  do  you  make  the  COEA  responsive  to  the 
changing  evolution  through  the  acquisition  milestone 
phases? 

•  How  can  the  Senior  Service  Colleges/Universities 
(NWC,  AWC,  NDU,  etc.)  contribute  to  the  acquisition 
process  by  enhancing  military  analysis  education? 

•  Contractor  support 

•  What  is  the  role  of  FFRDCs  in  COEAs? 

•  Should  contractors  do  COEAs? 

•  Should  vendors/bidders  do  COEAs? 

•  Management  Techniques? 

•  PERT 

•  POAMS 

•  TQM 

•  Who  should  sponsor/fund  COEAs? 

•  Which  organizational  structure  best  facilitates  the 
COEA  process? 

Working  Group  6:  Future  Directions 

Objective:  Define  and  report  on  the  role  of 

operations  research  in  the  COEA  process;  issues, 
problems,  and  research  areas,  and  the  future  directions 
for  military  operations  research  analysis  to  support 
COEAs  as  it  relates  to  this  working  group. 

Suggested  Issues: 

•  What  are  the  requirements  for  operations  research 
techniques  to  support  future  COEAs: 


•  Man-in-the-loop  simulation 

•  Data 

•  Cost  Effectiveness  Integration 

•  Milestone  Decision  Criteria 

•  What  analytical  techniques  are  appropriate 
for  each  Milestone  analysis? 

•  How  does  the  COEA  support  the 
definition  of  exit  criteria? 

•  Contractor  support 

•  What  is  the  role  of  FFRDCs  in  COEAs? 

•  Should  contractors  do  COEAs? 

•  Should  vendors/bidders  do  COEAs? 

•  What  additional  analysis  bear  on: 

•  Pre-Milestone  0  analysis 

•  Campaign  analysis 

•  Tests 

•  Linkage  to  other  program  documents:  MNS,  COEA, 
ORD,  TEMP,  etc. 

•  Incorporating  sensitivity  analyses  into  the  COEA: 

•  What  tools  are  available  to  accomplish  the 
sensitivity  analysis? 

•  How  do  you  make  the  COEA  responsive  to  the 
changing  evolution  through  the  acquisition  milestone 
phases? 
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4.2  Report  of  the  Test  and  Evaluation  Working  Group 


Issue:  How  can  the  OR  community  help  to  establish 
consistency  among  the  COEA  MOE/MOPs,  ORDs  and 
the  TEMP  and  relate  (Real  World,  DT  and  OT)  data  to 
the  COEAs? 

Discussion:  A  principal  issue  in  the  acquisition  process 
which  involves  the  T&E  community  is  how  can 
consistency  among  the  COEA  MOE/MOPs,  ORDs  and 
the  TEMPs  be  ensured.  An  equally  important  question 
is  how  can  pertinent  ("real  world,”  DT  and  OT)  data  be 
related  to  the  COEAs.  To  date,  there  are  too  many 
inconsistencies  and  lack  of  collaboration  in  preparating 
acquisition  documents.  In  addition,  problems 
frequently  arise  in  trying  to  incorporate  test  results  and 
cost  and  threat  changes  into  updating  COEAs. 

No  major  methodology  development  is 
envisioned  in  order  to  establish  and  solidify  this 
process.  Rather,  what  is  required  to  ensure  an  effective 
and  consistent  process  is  a  realignment  of  currently 
available  resources  and  possibly  a  reorganization  of 
personnel. 

In  regards  to  policy  implications,  what  is 
required  is  a  collaborative  approach  which  will 
establish  a  standardized  process  allowing  for  interaction 
among  all  players.  While  this  process  should  be 
specifically  laid  out,  it  must  be  flexible  and  certainly 
adaptable  to  a  changing  environment.  To  ensure  that 
this  process  does,  in  fact  occur  for  systems,  it  should  be 
incorporated  into  the  DoD  5000  series  andimplemented 
via  individual  Service  guidance. 

An  investigation  should  be  made  of  the 
feasibility  of  creating  linkages  within  the  COEA, 
TEMP  and  ORD  documents  which  ensures  crosswalks 
exist  between  these  documents.  Event-driven  updates 
of  the  COEA  throughout  the  acquisition  process  should 
be  implemented.  Although  the  DoD  5000  series 
currently  incorporates  COEA  updates,  if  required,  at 
each  milestone  review,  a  significant  change,  e.g. 
"threat,"  could  demand  an  update  of  the  COEA  between 
milestones.  A  formal  procedure  should  be  established 
to  provide  the  necessary  feedback  of  changes  to  threat, 
cost,  etc,  and  test  results  to  all  participants. 

An  immediate  step  should  be  taken  to  structure 
the  process  within  each  of  the  Services  to  ensure  that 
consistency  exists  among  the  COEA  MOE/MOPs, 
TEMP,  and  ORD  documents.  It  is  clear  that  in  the 


future  much  pressure  will  exist  to  achieve  consistency 
among  the  four  Services. 

The  actions  needed  to  attain  consistency  are  1) 
encouraging  a  feedback  process,  2)  setting  up  a  Test 
and  Analysis  Integration  Group  to  foster  and  oversee 
this  process  and  3)  obtaining  the  sanction  and  support 
of  OSD  and  Service  Principals. 

Issue:  How  can  Modeling  and  Simulation  techniques 
be  used  to  support  tests  and  COEAs? 

Discussion:  As  defense  budgets  decline,  and  more 
confidence  is  required  to  make  early  Milestone 
decisions,  one  approach  that  may  allow  us  to  increase 
confidence  to  an  acceptable  level  is  by  an  increased  use 
of  certified  modeling  and  simulation  (M&S). 

Many  M&S  tools  and  techniques  have  been 
used  by  the  defense  community  during  the  past 
decades,  including  linear  programming,  queueing 
theory,  Markov  chains  and  a  variety  of  Monte  Carlo 
simulations.  In  recent  years  newer  techniques  have 
been  developed  that  have  potential  to  increase  the  range 
and  fidelity  of  modeling  and  simulation  efforts.  Some 
of  these  include  distributed  interactive  simulations  (eg., 
SIMNET)  and  a  number  of  man  and/or  machine  in  the 
loop  techniques,  (e.g.,  TACCSF). 

Using  these  and  other  tools,  we  can  augment 
data  obtained  from  testing  by  using  M&S  to  interpolate 
between  testing  data  points  and  extrapolate  into  areas 
that  cannot  be  tested  (as  when  only  subcomponent  tests 
are  available  during  early  DT.)  As  examples,  M&S  can 
gather  data  on  operational  concepts,  reliability, 
availability,  maintainability,  and  sustainability.  M&S 
can  also  shed  light  on  logistics  and  training 
requirements  and  strategies,  as  well  as  generate 
information  on  notional  systems  that  do  not  yet  exist. 
Finally,  M&S  can  examine  the  integration  of  subsystem 
components  during  DT,  thereby  fulfilling  the  COEA 
requirement  to  assess  system  effectiveness  during  DT 
with  the  same  effectiveness  criteria  as  in  the  COEA. 

These  advancements  in  modeling  and 
simulation  now  enable  us  to  organize  models  into  a 
hierarchy  where  lower  resolution  models  feed  higher 
resolution  models  in  nearly  a  transparent  manner. 
However,  in  order  to  make  frail  use  of  this  advancement 
we  must  also  support  and  adopt  the  Defense  modeling 
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standardization  efforts  in  the  area  of  protocols  and  data 
elements. 

The  impact  of  modeling  and  simulation  on 
COEA  and  T&E  is  to  provide  a  wider  exploration  of  the 
"state  space."  In  addition  to  enabling  us  to  test  concept 

systems  that  don't  exist,  M&S  also  allows  us  to  explore 
conditions  and  scenarios  that  are  either  impossible, 
unsafe,  too  costly,  or  too  numerous  to  test.  This  does, 
however  emphasize  the  importance  of  model 
verification,  validation  and  accreditation,  ie. 
certification. 

We  feel  that  the  increased  use  of  M&S 
techniques  makes  the  new  acquisition  strategy  possible. 

As  the  new  acquisition  policy  is  being  formulated, 
it  seems  clear  that  early-on,  there  will  be  a  more  and 
stronger  reliance  on  the  use  of  M&S.  This  should  not 
be  construed  as  reduced  testing  or  necessarily  a 
substitution  of  modeling  for  testing,  but  rather  a 
complementary  program  of  testing  and  M&S  that  will 
ultimately  reduce  risk  in  the  acquisition  process. 


In  the  area  of  M&S  research,  we  recognized 
the  need  for  development  in  these  areas: 

-  C3I  (including  information  management) 

-  Dismounted  infantry  (some  work  has  been 
done  in  this  area) 

-  Artillery  suppression 

-  Countermeasures  (battlefield  smokes  & 
obscurants  and  battlefield  electronics) 

-  Inclusion  of  smart  weapons  into  battlefield 
scenarios 

-  Logistics  models  need  to  provide  the 
capability  for  design/cost  tradeoff 

-  Continued  development  of  new  S&/S 
techniques. 

Actions:  We  need  to  continue  to  develop  distributed 
interactive  simulations  and  other  research  tools.  We 
need  to  establish  a  standardized  and  accessible  database 
and  protocols.  We  need  to  continue  to  be  flexible  and 
proactivS. 
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MORS  COEA 

MINI-SYMPOSIUM 

MARCH  9-11,  1992 

TEST  &  EVALUATION  ISSUES 

TEST  &  EVALUATION 
WORKING  GROUP 

•  CONSISTENCY: 

How  Can  the  OR  Community  Help  to  Establish  Consistency 

Among  COEA  MOE/  MOPs.  ORD.  A  TEMPs  and  Relate 

Pertinent  -Real  World*  DT  &  OT  Data  to  the  COEAs 

•  LINKAGE: 

What  “Criteria'  Should  be  Used  to  Choose  Ways  of 

Establishing  The  Linkage  Required  by  DoD  5000  Series 

Regulations  That  Direct  That  MOEs  In  COEAs,  ORDs  And 

TEMPs  Be  Consistent  (i.e.  Linked) 

•  RISK: 

How  Does  one  Quantify  Risk  Associated  with  Limited 

Testing  of  Systems? 

CHAIRMAN 

John  Gehrig 

ODUSA(OR) 

•  IMS  Support. 

How  Can  M  &  S  Techniques  be  Used  To  Support  Tests  & 

COEAs? 

Figure  X:  Test  &  Evaluation  Woridng  Group  Flgure  2.  Tes,  &  Evalna(io„  bsues 


OBSERVATIONS  CONCERNING  CONSISTENCY 
OF  COEA  MOE/MOP  ORD  TEMP 


•  Currently  Inconsistencies  Exist  Among  COEA/ 
TESTS/ ORDS 

•  A  Colaborativa  Approach  to  Document  Preparation 
Is  Lacking 

•  Impediments  Exist  to  Incorporating  Test  Results, 
Cod  Changes,  and  Threat  Changes  Into  Updating 
COEAs 


Figure  3:  Observations 


Figure  4:  Consistency  Situation 


CONSISTENCY 
IMPACT  ON  ANALYSIS 

•  Realignment  of  Resources 

•  Reorganization  of  Personnel 

•  No  Major  Methodology  Development 


Figure  6:  Consistency  Impact 


Figure  5:  Consistency  Situation  Future 
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CONSISTENCY 

CONSISTENCY 

POLICY  IMPLICATION 

RESEARCH  AREAS 

•Develop  collaborative  approach: 

•Examine  the  feasibility  of  creating 

-Establish  a  standardized  process  for 

linkages  within  the  COEA,  TEMP,  ORD 

Interaction 

documents  which  ensure  crosswalks 

“Specific  yet  flexible  structure 

between  each,  (e.g.,  ORD  contains 

“Adaptable  to  changing  environment 

required  appendix  laying  out  MOEs 

•Incorporate  into  DOD  5000  series  and  Service 

for  COEA  and  TEMP 

implementation  guidance 

•  Require  event-driven  update  of  COEA 
throughout  acquisition  process. 

•  Establish  formal  procedure  to  feed-back 

changes  and  test  results  to  all  players. 

Figure  7:  Policy  Implication 


Figure  8:  Research  Areas 


CONSISTENCY 

ACTIONS 

•  Encourage  feed-back  AND 

modify  procedure 

•  Establish  a  test  and  analysis 

Integration  group 

-  Consideration  by 
-  OSD  Principals 
“  The  Services 


OBSERVATIONS  CONCERNING 
LINKAGE 


•  Dfleram  Approaches  Exist  to  Link  MOEs 
Among  COEA.  OR O.  &  TEMP 

-  Linkaga  Applies  Priinarity  lo  Operational  Effectiveness 

•  Documents  Developed  Concurrently  with  Precedence  Are 

1.  COEA 

2.  ORD 

3.  TEMP 


Figure  9:  Consistency  Actions 


Figure  10:  Observations 


CANDIDATE  LINKAGE  CRITERIA 

•  CREDIBILITY 

•  RESOURCEABILITY 

•  FLEXIBILITY 

■  MANAGEABILITY 


|  IN  OBOSt  OF  IMPORTANCE _ 

Figure  11:  Candidate  Linkage  Criteria 


LINKAGE 

IMPACT  ON  ANALYSIS 

•  ModeUSimulation  Accreditation 

•  Reactive  Analysis 

•  Audit/Traceability 


Figure  12:  Linkage  Impact  on  Analysis 
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LINKAGE 

POLICY  IMPLICATIONS 

- — - 

LINKAGE 

REASEARCH  AREAS 

•  Who  Does  Analysis  to  Choose  Linkage? 

•  Risk  Trade-off  Analysis 

•  When  is  Decision  Needed? 

•  Multiattribute  Analysis 

•  Who  Approves  Linkage? 

■  Linkage  Process 

1  . . 

Figure  14:  Linkage  Research  Areas 

Figure  13:  Linkage  Policy  Implications 


LINKAGE 

ACTION 

•  Develop  Subfactors/Measures/Techniques 
for  Rating  Each  Criterion 

•  Need  Implementation  Policy 


Figure  15:  Linkage  Action 


ASSUMPTIONS  CONCERNING  RISK 


•  Increased  time  for  development  does 

not  mean  more  testing 

•  Catch  22  applies.  Less  testing  means 

a  reduced  ability  to  defend  the 
program  politically 


Figure  16:  Assumptions  Concerning  Risk 


RISKS  OF  LIMITED  TESTING 
DEVELOPMENTAL  TESTING 

•  Inadequate  characterization  of  system  parameters 
•Weakness  of  data  makes  simulations  less  credible 

•  Failure  to  identify  hazards 

•  System  Integration  shortfalls 
•Software 

•  Unacceptable  performance  in  OT  /  after  fielding 
because  errors  cascade. 

Figure  17:  Risks  of  Limited  Testing 
Developmental  Testing 


- HISKSOFUMII LU  I bSTING 

OPERATIONAL  TESTING 

•  Does  not  get  exposed  to 
the  operational  environment 

•  May  not  resolve  suitability  criteria 

•  May  not  resolve  effectiveness  criteria 


Figure  18:  Risks  of  Limited  Testing 
Operational  Testing 
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QUANTIFICATION  OF  RISK 

•  CONSIDERED: 

RISK-  probability  (failure)  X  Consequences  (failure) 

•  BUT  REJECTED  BECAUSE 

•  Too  hard 

•  Lots  subject  to  reliable  sensitivity  analysis 

Figure  19:  Quantification  of  Risk 


Contributions  of  Simulation 


Figure  20:  Contributions  of  Simulation 


Challenge 

RISK 

IMPACT  ON  COEA 

-  Ate-r.te.lte—  - - 

/ 

•  Program  costs  may  be  higher  AND 

/ 

may  delay  deployment  of  important 

c 

0 

systems. 

f 

- 

•  Confidence  in  operational 

d 

How  do  we  quantify 

effectiveness  reduced 

• 

n 

/ 

this  line? 

•  RDT&E  costs  may  be  transferred  to 

c 

/ 

• 

O&M  with  probable  increase  in 

Ufa  cycle  costs 

T  ■  Wl 

Teetins  Twang 

All 

abnuMon 

•Confidence  in  COEA  itself  is  reduced 

Figure  21:  Challenge  Figure  22:  Risk  Impact  on  COEA 


RISK 

OTHER  CONSIDERATIONS 

*  Policy  impBccfons 

•  Adequate  testing  b  essential  AND 
It  must  correlate  and  support  analysis 

•  Research  areas 

•Historical  analysis  to  determine  tow  much 
teeing  b  enough 

Tradeoff  between  tasting  and  analysis  activities 

ACTION 

MORS  to  Implement 

(recommend  AF  Academy  Feb  or  March) 


Figure  23:  Other  Considerations 


M&S  CONSIDERATIONS  FOR 
TESTS  &  COEAs 

•  New  M&S  T echnlques  are  Available 
-DIS 

-  Man/Hardware  in  the  Loop 

•  M&S  Can  Augment  Data  Obtained  From  T&E  on: 

-  Operational  Concepts 

-  RAM  &  Sustainability 

-  Logistice  Requirements 
-Training  Requirements 

•  Can  Also  Anticipate  impact  of  Proposed  Systems 

•  Models  Must  be  Organized  in  a  Hierarchy  with 
Standardized  Accessible  Data  Base 


Figure  24:  M&S  Considerations 
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M&S  IN  SUPPORT  OF 
TESTS  &  COEAs 


•  Impact  of  M&S  on  T&E 

-  Wider  Exploration  of  "State  Space" 

-  Model  Validation  Critical 

•  Policy  Implications 

-  Makes  New  Acquisition  Strategy  Possible 

•  Research 

-  C3I 

-  Dismounted  Infantry 

-  Artillery  Suppression 
-CM 

-  Smart  Weapons 

•  Actions 

•  Develop  Data  Base 

-  Develop  DIS  &  other  Research  Areas 

-  Flexible  &  Proactive 


Figure  25:  M&S  in  Support  of  Tests  &  COEAs 
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4.3  Report  of  the  Threat  Assessments  and  Scenarios 
Working  Group 

Overview 

The  backdrop  or  background  to  the  Defense 
Acquisition  Board  process  is  the  threat  environment  in 
which  a  proposed  weapon  system  will  operate.  During 
the  development  cycle,  the  threat  will  change, 
predictions  of  future  threats  will  change  and  what  we 
think  we  know  about  the  threat  will  change.  Threat 
assessments,  scenarios,  the  COEA  process  and  the 
operations  research  community  must  react  and  respond 
to  these  changes  to  present  the  effectiveness  and 
capabilities  and  vulnerabilities  of  the  weapon  system  as 
accurately  as  possible.  There  are  measures  one  can 
take  to  minimize  and  respond  to  changes  during  the 
acquisition  process. 

Issues 

Working  Group  2:  Threat  Assessments  and 
Scenarios  discussed  and  chose  to  address  three  issues 
or  areas  of  concern:  the  dynamic  threat,  level  of  detail, 
and  analysis  of  operational  effectiveness  with  scenarios. 
We  have  highlighted  impacts  on  analysis  and  impacts 
on  research.  This  area  is  a  broad,  diverse  area  and  we 
provide  some  general  guidelines  and  suggestions  for 
your  consideration. 

Dynamic  Threat 

In  today's  environment,  we  may  expect  the 
threat  to  continue  to  change.  These  changes  may  not  be 
as  dramatic  as  the  changes  that  have  occurred  over  the 
past  two  years.  However,  the  threat  is  dynamic,  and 
change  is  inevitable.  Threat  changes  may  occur  at  any 
time,  either  during  the  drafting  of  a  COEA  or  after  the 
completion  of  a  milestone  COEA.  If  the  changes  to  the 
threat  are  significant,  the  validity  of  a  COEA  may  be 
called  into  question. 

To  account  for  rapid  threat  changes  during  the 
COEA  development,  COEA  analysts  are  faced  with  a 
number  of  possible  decisions.  First,  the  COEA  analyst 
may  react  to  changes  by  making  rapid  modifications  to 
the  COEA  while  continuing  to  meet  the  schedule.  This 
"scramble  and  panic"  mode  is  all  too  often 
representative  of  many  of  today's  COEA  processes. 
Second,  if  the  changes  in  a  threat  are  considered  to  be 
sufficiently  small,  localized,  or  insignificant  to  warrant 
a  major  COEA  update.  In  addition,  "freeze  the  threat" 
is  an  option  to  consider.  While  such  a  freeze  may  make 


the  COEA  dated,  if  the  COEA  analyst  were  to 
document  his/her  rationale  then  the  COEA  may  still 
serve  as  a  useful  historical  document.  Third,  if  critical 
threat  information  is  unavailable  or  is  subject  to 
frequent  change,  then  COEA  analysts  must  conduct 
sensitivity  analyses  and  parametric  studies  to  determine 
the  impacts  of  threat  changes  on  COEA  results.  In  all 
cases,  sensitivity  analysis  and  parametric  studies  add  to 
flexibilitiy  in  a  COEA. 

COEAs  need  to  plan  for  change.  Due  to  threat 
and  other  changes,  COEA  planning  should  be  flexible 
to  incorporate  changes.  New  tools,  methodologies,  and 
techniques  need  to  have  flexibility  designed  into  them 
up-front.  To  ensure  flexibility  and  responsiveness, 
periodic  in-progress  reviews  should  be  conducted 
throughout  the  COEA  process. 

Threats  may  also  change  after  a  milestone 
decision  has  been  made.  In  this  case,  the  change  must 
be  reconsidered  under  the  above  criteria  (impact  size, 
localization,  and  significance)  to  see  if  an  update  is 
warranted.  If  warranted,  an  updated  COEA  should  be 
produced  expeditiously  and  results  should  be  provided 
to  principle  decision  makers  as  soon  as  possible.  The 
original  COEA  should  be  treated  as  a  baseline  for 
future  comparisons.  If  an  updated  COEA  is  warranted, 
then  that  COEA  should  incorporate  the  complete  set  of 
changes  that  have  been  identified  to  date  —  including, 
threat,  requirements,  scenarios,  etc. 

Level  of  Detail 

Due  to  the  nature  of  the  COEA  process,  the 
level  of  detail  which  needs  to  be  considered  increases 
for  successive  milestones.  The  emphasis  for  Phase  0  is 
to  be  flexible  enough  to  encompass  the  attributes  of 
multiple  concepts  which  may  negate  the  operational 
deficiencies  identified  in  the  mission  needs  statement. 
For  phase  1 ,  the  COEA  must  be  more  detailed  so  as  to 
identify  performance  differences  between  specific 
solutions  identified  as  being  capable  of  meeting  the 
mission  needs  statement  operational  shortfalls.  Updates 
of  the  COEA  may  be  needed  during  phase  1  or  later 
phases  to  account  for  changes  in  understanding  of  the 
threat,  advances  in  US  technology  base,  and/or  later 
cost  data. 

The  specificity  of  the  mission  needs  shortfalls 
require  a  matching  level  of  threat  input  parameters. 
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Therefore,  a  mission  needs  statement  which  addresses 
a  very  specific  need  may  require  a  more  detailed  threat 
definition  early. 

Current  US  military  and  political  viewpoints 

suggest  that  the  COEA  utilize  an  integrated  analysis  to 

take  into  account  the  effect  of  proposed  systems  on  the 
outcome  of  the  joint  campaigns.  The  effectiveness  of 
the  proposed  systems  should  be  analyzed  by  examining 
all  aspects  of  joint  operations. 

Analyze  Operational  Effectiveness  With  Scenarios 

Initially,  basic  assumptions  contained  in  the 
COEA  must  be  clearly  defined.  These  would  include 
assumptions  about  the  threat  and  operational  concepts. 
This  will  form  a  baseline  for  understanding  future 
milestone  objectives  and  changes.  The  scenarios  must 
be  analytically  broad  enough  to  address  requirements  of 
the  Mission  Need  Statements  and  to  explore  the  range 
of  conflicts  or  environments.  Include  aspects  of  joint 
operations  in  which  a  weapon  system  may  be 
employed.  The  range  and  scope  of  these  scenarios  need 
to  be  tailored  and  will  probably  be  unique  to  each 
COEA.  Scenarios  should  be  sufficiently  robust  to 
capture  the  differences  in  effectiveness  and  the  cost 
implications  of  each  alternative.  The  scenarios  should 
also  allow  exploration  of  technology,  employment,  and 
tactics  of  opposing  systems,  either  as  excursions  or  in 
sensitivity  analyses. 

Impact  on  Analysis 

Scenarios  must  be  able  to  withstand  the  "what 
if'  challenge.  They  must  challenge  the  alternatives 
being  considered. 

The  analyst  should  be  careful  to  avoid 
focusing  on  what  he  understands  versus  what  is  critical. 
This  kind  of  bias  can  ignore  critical  aspects. 

Robustness  is  necessary  in  threat  analysis,  in 
the  derived  scenario,  and  in  the  model.  A  valid  test  of 
robustness  is  achieved  when  variations  in  threat  and 
model  parameters  result  in  consistent  and  predictable 
results  that  do  not  alter  the  preference  rankings  of  the 
alternatives. 

The  models  often  require  more  specific  detail 
than  is  generally  available  -  the  question  is,  how  do  we 
get  it?  Threat  details  should  be  supplied  by  the 
intelligence  community.  Other  details  need  to  be 
created  by  the  analyst,  within  reasonable  bounds  and 


while  maintaining  credibility.  Study  teams  help  bring 
a  level  of  credibility  and  rationality  to  this  process. 

Campaign  scenarios  should  address  plausible 
deployment  and  employment  timelines.  For  many 
weapons  systems,  the  ability  to  respond  rapidly  to  a 

theater  commander's  needs  is  as  important  as  its 
capability  in  combat.  Similarly  the  ability  to  conduct 
sustained  operations  over  time  -  the  logistics  and 
reliability  dimensions  of  a  candidate  weapons  system  is 
a  necessary  measure  of  capability.  Considering  all 
phases  of  a  theater  operations  will  allow  you  to 
accomplish  this  dimension. 

Analytical  resources  need  to  be  allocated  to 
explore  the  sensitive  and/or  interesting  facets  of  the 
theater  operation  timeline.  This  may  require  extreme 
detail  in  some  areas  while  giving  cursory  analysis  to  the 
less  relevant.  If  this  is  the  case,  insure  you  have 
documented  the  logic  for  your  methodology. 

The  analyst's  dilemma:  designing  flexibility 
into  a  scenario  is  a  problem  because  of  the  continuous 
changes  in  the  threat  environment.  Scenarios  must  be 
designed  so  as  to  accomodate  changes  in  the  threat 
throughout  the  evolution  of  the  developing  system. 
This  flexibility  should  be  included  so  that  to  the 
greatest  extent  possible,  the  scenario  retains  those 
attributes  which  derived  the  original  requirements  for 
the  system. 

To  assure  that  the  needs  of  various 
communities  within  the  acquisition  process  are 
recognized,  and  that  the  capabilities  of  these  various 
communities  to  contribute  to  the  COEA  process  are 
exploited,  it  is  critical  that  close  working  relationships 
among  the  communities  be  established  early  in  the 
COEA  effort. 

Impact  on  Research 

There  is  an  increased  requirement  for 
assessments  of  a  historical,  current,  and  projected 
nature.  A  shift  from  a  European  conflict  to  a  variety  of 
regional  conflicts  necessitates  assessing  a  host  of  new 
potential  conflicts  and  technologies.  This  should  be 
done  in  an  historical/current  context  in  order  to  make 
reasonable  projections  regarding  potential  operational 
environments,  capabilities,  and  intentions. 

There  is  a  need  to  expand  the  data  base  to 
address  more  fully  the  characteristics  of  Blue,  Red,  and 
Gray  systems.  In  this  context,  blue  refers  to  US 
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systems;  red  to  systems  in  the  inventory  of  hostile 
forces;  and  gray  systems  normally  non-US  systems 
available  to  both  potentially  hostile  as  well  as 
coalition  forces. 

On-line  availability  of  these  data  would 
improve  currency  and  completeness  of  the  effectiveness 
analysis. 

There  is  also  a  need  to  develop  methods  of 
comparing  the  results  of  different  models  used  to 
analyze  different  alternatives.  Especially  as  more 
emphasis  is  placed  on  joint  operations.  For  example,  in 
evaluating  the  contributions  of  alternative  ground-based 
and  air-based  systems  may  require  the  integration  or 
comparison  of  results  from  various  models. 


There  is  a  need  for  a  certified  listing  of 
models,  databases  and  scenarios  for  use  in  COEAs. 
This  will  ensure  a  thread  of  commonality  and 
consistency  when  comparing  COEA  results.  This 
allows  decision-makers  a  reasonable  logic  frail  for 
program  trade-offs. 

There  is  a  need  for  distributed/networked 
analysis  capability  between  centers  of  expertise  or 
excellence.  This  will  result  in  broader,  more  detailed, 
credible,  and  consistent  analytic  results.  It  will  help 
reduce  redundancy,  save  resources,  improve  quality, 
and  facilitate  time-savings. 
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OVERVIEW 

•  ISSUES/  DISCUSSION  POINTS 

•  IMPACT  ON  ANALYSIS 

•  IMPACT  ON  RESEARCH 
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ISSUES 


•  DYNAMIC  THREAT 

•  LEVEL  OF  DETAIL 

•  ANALYZE  OPERATIONAL  EFFECTIVENESS 
WITH  SCENARIOS 


DYNAMIC  THREAT 


•  THREAT  CHANGES  DURING  IN  PROGRESS  COEAS 

-  FREEZE  THE  THREAT 

-  FLEXIBILITY 

-  DESIGN  IN  UP  FRONT 

-  HOLD  IN  PROGRESS  REVIEWS 

-  SENSITIVITY  ANALYSIS 

-  UPDATE  COEA  (TIMELINES) 

•  THREAT  CHANGES  AFTER  MILESTONE 

-  UPDATE  COEA 

-  ORIGINAL  WORK  IS  BASELINE 

-  INCORPORATE  ALL  CHANGES  (THREAT,  REQUIREMENTS, 
SCENARIOS,  ETC) 
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LEVEL  OF  DETAIL 


•  DETAILS  WILL  INCREASE  FOR  SUCCESSIVE  MILESTONES 

-  PHASE  0  COEA  FLEXIBLE  (MULTIPLE  CONCEPTS) 

-  PHASE  1  COEA  MORE  DETAILED  (SPECIFIC  SOI  JJTION(S)) 

-  UPDATES  WILL  REMAIN  DETAILED 


•  MISSION  NEED  DRIVES  DETAILS 


•  INTEGRATED  APPROACH/  JOINT  CAMPAIGN  ANALYSIS 
-  EXAMINE  ALL  ASPECTS  OF  JOINT  OPERATIONS 


TESTING  OPERATIONAL  EFFECTIVENESS 
WITH  SCENARIOS 

•  THE  OPERATIONAL  EFFECTIVENESS  OF  A 
SYSTEM  IS  BEST  EXAMINED  THROUGH  A 
SERIES  OF  SCENARIOS 


•  SCENARIOS  MUST  ADDRESS  THE  RANGE  OF 
PLAUSIBLE,  POTENTIAL  CONFLICT 
ENVIRONMENTS 

-  EXPLORE  HIGH,  MEDIUM,  AND  LOW 
INTENSITY  CONFLICTS  (  QUANTITY/ 
QUALITY) 

-  THE  APPROACH  WILL  BE  UNIQUE  TO 
EACH  COEA 
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REGIONAL/  CONTINGENCY  THREAT  SCENARIOS 


•  CLEARLY  DEFINED  BASIC  ASSUMPTIONS 

•  IDENTIFY  POLITICAL  AND  ECONOMIC  FACTORS 

•  EXPLORE  TECHNOLOGY  LEVEL  OF 

OPPOSING  SYSTEMS 

•  JOINT  CAMPAIGN  ANALYSIS 

-  SYNTHESIS/  EXAMINE  ALL  ASPECTS 

OF  JOINT  OPERATIONS 

•  ROLE  OF  BLUE  FORCES 

-  EXPLORE  ALTERNATIVE  OUTCOMES 


BROAD  ANALYTICAL  SCENARIOS 


•  SHOULD  BE  CAPABLE  OF  BEING  MODELED  ANALYTICALLY 

•  NEED  TO  ADDRESS  MISSION  NEEDS  STATEMENT  REQUIREMENTS 

•  INITIALLY  DETAILED  AND  EXPANSIVE;  NARROW  LATER 

•  BE  ABLE  TO  CAPTURE  DIFFERENCES  AND  ALTERNATIVES 

•  TAILOR  TO  MEET  COEA  OBJECTIVES 

•  NOT  GEOPOLITICALLY  ORIENTED 

•  SCENARIOS  SHOULD  ADDRESS  ALL  NECESSARY  PHASES  OF 
OPERATIONS 


IMPACT  ON  ANALYSIS 


ASSESS  ABILITY  OF  SCENARIOS  TO  ADDRESS  QUESTIONS 

FOCUS  ON  CRITICAL  ASPECTS  OF  SCENARIOS 

INCLUDE  EXCURSIONS  TO  BETTER  EVALUATE 
ROBUSTNESS 

SPECIFY  DETAIL/  DATA  REQUIRED  FOR  BOTH  RED 
AND  BLUE 

DECIDE  WHICH  PHASES  OF  OPERATION  ARE  REQUIRED 
DO  PHASES  AT  REQUIRED  LEVEL  OF  DETAIL 
DESIGN  IN  FLEXIBILITY  (I.E.  SENSITIVITY  ANALYSIS) 
DEVELOP  CLOSER  WORKING  RELATIONSHIPS 


IMPACT  ON  RESEARCH 


INCREASED  REQUIREMENT  FOR  ASSESSMENTS  OF  A  HISTORICAL, 
CURF“NT,  AND  PROJECTED  NATURE 

DATA  DEVELOPMENT  WILL  ADDRESS  BLUE/RED/GRAY  SYSTEMS 

MEANS  TO  COMPARE  RESULTS  FROM  DIFFERENT  MODELS  TO 
COMPARE  DIFFERENT  ALTERNATIVES 

DATABASE  OF  CERTIFIED  MODELS,  APPROVED  THREATS  AND 
SCENARIOS 

DISTRIBUTED/  NETWORKED  JOINT/  INTEGRATED  ANALYSIS 
CAPABILITIES  -  CENTERS  OF  EXCELLENCE 
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ANALYZE  OPERATIONAL  EFFECTIVENESS 
WITH  SCENARIOS 

•  CLEARLY  DEFINE  BASIC  ASSUMPTIONS 

•  BROAD  ANALYTICAL  SCENARIOS  NEEDED  TO 
ADDRESS  MNS  REQUIREMENTS 

-  EXPLORE  RANGE  OF  CONFLICTS/  ENVIRONMENTS 

-  UNIQUE  TO  EACH  COEA 

•  CAPTURE  DIFFERENCES  OF  ALTERNATIVES 

•  EXPLORE  TECHNOLOGY  OF  OPPOSING  SYSTEMS 

-  EMPLOYMENT  TACTICS 


4.4  Report  of  the  Effectiveness  Methodologies 
Working  Group 

Working  Group  3  was  charged  with  looking  at 
issues,  problems,  and  research  topics  within  the  area  of 
effectiveness  methodologies.  This  area  was  viewed  as 
having  two  components:  precursor  activities  (front  end 
analysis)  and  effectiveness  tools.  The  "front  end"  of  the 
effectiveness  analysis  must  clearly  define  the  problem, 
establish  the  approach,  and  select  the  appropriate  set  of 
metrics.  Knowing  where  you  are  going  and  how  to  get 
there  is  both  challenging  and  critically  important  to  a 
successful  COE  A. 


SETTING  UP  THE  ANALYSIS 
Defining  the  Problem 

Each  COEA  has  a  unique  set  of  concerns  that 
the  ultimate  decision  makers  require  the  analysts  to 
address.  Defining  the  problem  is  the  process  by  which 
these  unique  concerns  are  identified,  described,  and 
understood.  The  process  must  enable  the  COEA  team 
to  internalize  the  decision  maker's  needs,  where  the 
program  is  in  the  acquisition  process,  and  the  decisions 
to  be  made. 

Defining  The  Approach 

Once  there  is  a  through  understanding  of  the 
problem,  the  team  should  define  and  establish  the 
overall  approach.  Included  within  this  step  is  a  group 
of  interrelated  and  iterative  activities.  One  activity  is 
the  identification  of  alternatives  which  may  be  given  to 
the  team  by  the  sponsor  or  expanded  by  the  team  to 
consider  all  appropriate  solutions.  The  approach 
selected  must  identify  and  consider  key  analysis 
questions,  the  essential  elements  of  analysis  (EEA). 
The  EEA  represent  that  set  of  questions,  which  when 
answered,  will  completely  address  the  issues  associated 
with  the  COEA  decision.  Lastly,  the  approach 
definition  step  must  settle  issues  of  methodology, 
assumptions,  and  constraints. 

Selecting  Metric 

The  level  of  analysis  is  a  function  of  the  phase 
of  the  acquisition  process.  At  each  stage,  the  decision 
maker  has  a  defined,  evolved,  set  of  missions/objectives 
that  must  be  satisfied.  The  metrics  need  to  assess  how 
well  the  alternatives  achieve  those  objectives. 
Qualitative  and  quantitative  metrics  should  be 


examined.  The  metrics  selected  must  be  meaningful  to 
the  audiences  to  which  the  COEA  must  answer.  These 
audiences  include  not  only  the  DOD  Programmers,  but 
also  the  Congress,  the  Executive  Branch,  and  the 
CINCs. 

Process  Issues 

A  well  understood  and  accepted  COEA 
process  must  be  institutionalized  to  ensure  consistent 
results  in  the  face  of  changing  leadership,  issues  and 
tools.  Relying  upon  the  personality  of  connections  of 
an  individual  study  director  is  a  hit  or  miss  proposition 
at  best  and  does  not  capture  the  processes  required  to 
ensure  a  good  analysis  effort.  Communication  between 
the  decision  maker  and  the  producer  of  the  COEA 
document  that  supports  the  decision  making  process  is 
inadequate. 

Formal  communication  is  provided  as  a  result 
of  an  ADM  followed  by  a  rather  lengthy  period  (18 
months)  with  no  formal  communication.  This  makes 
the  COEA  dependent  on  the  Study  Director's 
capabilities  in  seeking  informal  guidance.  This  can 
cause  wide  variance  in  the  COEA's  usefulness  to 
decision  makers.  Typically  the  COEA  team  is  both 
physically  and  organizationally  remote  from  the 
decision  makers.  This  impacts  on  the  study  directors 
access  to  decision  makers  for  in  process  review.  This 
impact  manifests  itself  as  intermediate  command 
guidance  on  the  questions  the  COEA  is  to  address,  and 
a  screening  of  COEA  team  requests  for  guidance  from 
the  decision  makers.  No  direct  paths  exists  between 
the  COEA  team  and  the  decision  maker  (or  his 
representative)  to  keep  the  COEA  properly  focused. 
Recent  experience  demonstrates  that  over  the  normal 
COEA  time  frame  that  the  environment  changes  rapidly 
for  some  programs.  Leadership  and  environmental 
changes  affect  National  goals  and  policies.  These  in 
turn  result  in  changes  in  military  objectives  and  the 
content  of  programs  needed  to  accomplish  these 
objectives.  These  changes  must  be  incorporated  in  the 
COEA  process  to  enable  the  decision  maker  to  make 
valid  decisions. 

Lessons  Learned 

Currently  the  procedures  to  leam  how  to 
conduct  a  COEA  are  not  imbedded  either  in  the  service 
academies  processes  or  in  the  informally  supported 
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methods  characterized  by  lessons  learned  from  media, 
symposia,  etc.  The  inclusion  of  an  exit  report  from  the 
director  of  the  study  into  and  accessible  data  base  to 
preserve  institutional  memory  for  future  study  directors 
would  be  and  approval.  Included  should  be  the 
methodology  that  was  followed  and  any  that  were 

considered  and  disregarded.  Rationale  as  to  why  a 
specific  procedure  was  used  as  opposed  to  another 
needs  to  be  included.  Measures  used  to  establish  the 
merit  of  subjects  of  study  should  be  captured  and 
summarized  for  potential  future  use.  A  "Roadmap" 
needs  to  be  established  to  enable  the  COEA  to  achieve 
a  better  chance  of  success.  All  the  services  train  for 
success  in  war,  they  also  need  to  train  for  success  in  the 
peacetime  warfare  associated  with  doing  a  COEA. 
Positive  feedback  to  current  study  doers  is  a  useful 
procedure  to  reinforce  successful  efforts.  A  mechanism 
to  do  this  needs  to  be  included  in  the  COEA  process. 

Continuity 

Timeliness  involved  with  the  COEA  cycle,  to 
include  the  updates  involved  at  different  milestones 
stretch  the  ability  of  the  study  doer  to  maintain 
continuity  of  effort.  Recent  changes  in  Europe  are 
evidence  of  the  changing  context  in  which  COEA 
efforts  are  coordinated.  Along  with  the  threat 
implications,  the  services  have  doctrinal,  strategic  and 
operational  issues  that  change  over  time.  These  often 
need  to  be  reflected  in  the  study  proceeding  or  just 
ended.  Accomplishing  the  task  requires  initial  planning 
flexibility  to  absorb  the  additional  requirements, 
changes  r  direction  or  evaluation  of  sensitivity  analysis 
that  war  .ginally  not  anticipated. 

Research  Issues 

A  second  set  of  issues  deals  with  areas  in 
which  the  tools  available  to  support  effectiveness 
analysis  are  inadequate.  This  area  naturally  suggests 
research  to  improve  existing  tools  or  develop  new  ones. 

The  first  area  focuses  on  the  approaches 
available  to  the  OR  community  and  their  relevance  and 
transparency  to  the  ultimate  consumer  —  the  decision 
maker.  Many  of  the  tools  available  may  be  so  esoteric 
that  they  are  either  not  understood  by  or  not  accepted 
by  the  decision  makers.  On  the  other  hand,  many 
analysts  do  not  appreciate  fully  either  the  issues  the 
decision  makers  need  to  address  or  their 
perspective/logic.  A  key  need  in  developing  the 
approach  is  to  assure  that  the  methodology  selected 


focuses  on  the  criteria  to  be  used  in  making  the 
decision. 

The  second  area  deals  with  the  whole  issue  of 
the  identification  and  selection  of  appropriate  measures. 
In  the  changing  threat  and  budgetaiy  environment,  it 

may  well  be  that  the  traditional  measures  (attrition, 
FLOT  movement,  exchange  ratios,  etc.)  no  longer 
apply.  There  is  a  need  to  develop  agreed  upon 
measures  for  non-attrition  warfare,  indeed,  for  the  full 
range  of  conflicts  most  likely  in  the  future.  There  is  also 
the  need  to  develop  a  hierarchy  of  measures  that  allow 
you  to  relate  campaign  outcomes  to  MOEs  and  MOEs 
to  MOPs.  It  is  the  MOPs,  at  the  system  level,  that 
provide  the  linkage  between  the  analysts  and  the  testers. 

Finally,  many  of  the  problems  will  be 
qualitative  in  nature  and  therefore  there  is  a  need  to 
develop  an  agreed  upon  set  of  qualitative  measures. 
We  expect  fewer  weapon  systems  new  starts  and  more 
supporting  systems,  including  C3I  and  mobility 
enhancements,  new  starts.  Measures  are  needed  to 
allow  us  to  address  the  performance,  effectiveness,  and 
combat  contributions  of  these  latter  systems.  Even  in 
the  case  of  weapon  systems,  qualitative  measures  are 
often  appropriate  and  meaningful  to  the  decision 
makers. 

The  OR  community  is  the  logical  venue  to 
develop  new  tools,  not  only  to  resolve  recognized 
deficiencies  but  also  to  take  advantage  of  new 
techniques,  theories,  and  approaches. 


KEY  RECOMMENDATIONS 

Take  advantage  of  modem  technology  to 
improve  the  inadequacies  of  the  COEA  process.  Teach 
the  COEA  community  how  to  do  a  successful  COEA. 

We  need  to  create  a  program  to  resolve  the 
COEA  process  issues  as  they  occur.  This  can  be 
implemented  in  three  ways.  First,  create  an  interactive 
library/data  base  of  COEA  lessons  learned.  This  data 
base  contains  metrics  that  describe  successful  and 
unsuccessful  COEAs.  Next,  continue  with  COEA 
symposia  to  get  lessons  learned  out  to  the  COEA 
community'.  Finally,  establish  networks  of 

specialists/identify  leadership. 

We  need  to  research  innovative  methods  to 
bring  OR  decision  making  methods  to  the  decision 
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makers.  We  have  to  make  the  process  and  results  more 
understandable  to  the  decision  maker.  We  can  do  this 
by  making  use  of  modem  technology  to  involve  the 
decision  maker  in  the  analysis  process.  This  may 
require  working  with  the  decision  maker. 

We  need  to  do  research  to  define  types  of 
measures  that  are  appropriate  for  the  emerging  conflict 
environments.  The  past  will  not  necessarily  be  a  good 
predictor  of  future  measures.  We  are  moving  toward 
Joint  Service  and  Combined  Allied  operations.  Also, 
non-weapon  systems  can  be  key  to  influencing  the 
outcome  of  the  conflict.  Campaign  analysis  in  the 
future  will  probably  involve  limited  objectives  and  may 
require  different  measures  than  the  past,  more 
traditional  conflicts. 


EFFECTIVENESS  METHODOLOGIES 
Introduction 

The  challenge  facing  this  working  subgroup 
was  significant  for  it  accepted  the  task  of  defining  and 
outlining  future  COEA  effectiveness  methodologies. 
As  expected,  however,  most  of  the  discussions  focused 
on  member's  experiences  using  current  tools  and 
methods  in  addressing  COEA  issues.  When  asked  to 
extend  their  experiences  to  future  requirements  the  real 
challenge  surfaced.  In  order  to  produce  a  useful 
product  in  the  short  time  allotted  at  the  symposium  the 
subgroup  limited  its  discussions  to  the  following  two 
topics; 

(1)  Develop  insight  into  future 
methodologies  through  review  of  the  decision 
making  process  and  issues  for  COEA 
effectiveness  modeling. 

(2)  Provide  insights  into  practical 
"campaign  analysis"  modeling  techniques. 

General 

The  concept  of  COEA  effectiveness  analysis 
is  not  new  but  the  current  climate  of  military  decision 
making  does  present  an  environment  different  from  the 
past  decade.  COEA  effectiveness  analyses  are  now 
used  within  a  decision  making  environment  with 
several  new  characteristics  which  places  considerable 
strain  on  many  of  the  current  modeling  tools.  For  the 
past  several  decades  the  following  model  environmental 
parameters  have  been  reasonably  stable;  a  dominant 


threat,  analytical  requirements  focused  on  single  service 
priorities  and  acquisition  dollars  were  sufficient  to 
allow  each  service  several  major  programs.  As  a  result 
of  significant  changes  in  all  of  these  parameters  the 
analytical  community  is  faced  with  the  challenge  of 
developing  modeling  tools  for  this  new  era.  All  of 
these  factors  imply  a  demonstrative  change  in  modeling 
tools  to  effectively  address  future  effectiveness  issues. 

COEA  work  of  the  future  will  take  a  joint  and 
possibly  combined  perspective  and  cover  a  broader  set 
of  scenarios  and  issues.  The  focus  of  "campaign 
analysis"  is  designed  to  extend  traditional  COEA 
analysis  into  a  more  general  and  useful  process  for  the 
service  and  OSD  decision  maker.  A  caution  must  be 
sounded  because  of  the  tendency  to  link  COEA 
effectiveness  analyses  with  affordability  and 
programmatic  issues.  The  subgroup  as  well  as  the 
senior  analyst  panel  felt  that  these  two  sets  of  issues, 
effectiveness  and  affordability,  should  remain  separate. 
These  two  sets  of  issues  are  closely  related  and  must  be 
complementary  but  not  so  dependent  that  they  lose  their 
distinction.  A  close  examination  of  the  measures  of 
performance  and  essential  elements  of  analysis  in  each 
of  these  two  arenas  shows  that  there  are  distinctive 
dynamics  in  each  and  that  effectiveness  should  not  be 
dependent  upon  affordability  issues. 

Modeling  Environment 

It  is  critical  to  start  the  model  development 
process  at  the  beginning!  Discard  preconceived  notions 
of  appropriate  techniques  and  methods  and  allow  the 
systems  analysis  process  and  the  problem  statement  to 
develop  appropriate  tools  and  techniques.  This 
paradigm  will  be  the  key  to  future  modeling 
developments.  With  this  thought  the  subgroup  first 
examined  the  environment  in  which  effectiveness 
methodologies  will  be  used.  Immediately,  the  subgroup 
saw  that  the  problem,  COEA  effectiveness  analysis  and 
associated  campaign  analyses,  is  not  sufficiently 
defined  to  focus  directly  on  one  modeling  technique. 
What  are  COEA  decision  issues?  Are  they  identical  to 
campaign  analysis  decision  issues?  Are  one  set  of 
issues  and  requirements  a  subset  of  the  other?  How 
dependent  should  the  COEA  analysis  be  upon  the 
campaign  analysis?  (Prof  Hughes  offers  his  opinion  on 
this  last  question  in  his  comments.)  Additionally,  it 
was  difficult  to  forecast  either  the  military  scenario  and 
forces  associated  with  either  a  COEA  or  the  more 
robust  campaign  analysis.  Effectiveness  measures  of 
past  COEAs  may  satisfy  some  future  issues,  but  there 
is  a  strong  likelihood  that  new  issues  will  arise. 
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The  old  axiom;  don't  make  the  tool  fit  the 
problem  rather  let  the  problem  define  the  appropriate 
tool,  should  now  be  restated  for  we  may  tend  to  select 
the  method  of  solution  before  we  know  the 
problem/issue.  To  be  most  effective  the  methodology, 

and  model  if  one  is  used,  should  address  the  decision 
issues  as  directly  as  possible.  It's  because  of  this  single 
facet  of  the  future  analytical  environment,  uncertainty 
in  issues  and  scenarios,  that  a  single  modeling 
technique  may  fall  well  short  of  satisfying  future 
modeling  requirements.  Uncertainty  both  in  model 
requirements  and  analytical  issues  forces  the  analyst  to 
remain  flexible  in  his  approach  to  COEA  problem 
solving.  It  is  ill  advised  to  presume  that  a  single  model 
designed  with  rather  severe  limitations  in  scope  and 
flexibility  will  serve  future  COEA  analytical  needs. 

Despite  uncertainty  the  characteristics  and 
nature  of  future  modeling  requirements  do  provide 
useful  guidance  in  developing  appropriate  tools  and 
methods.  What  are  characteristics  of  future  issues  and 
what  are  the  factors  that  will  influence  scenario 
development?  There  are  several  factors  that  influence 
and  strengthen  the  need  for  a  flexible  and  tailored 
process;  shorter  analytical  cycles,  broad  spectrum 
analysis,  non-threat  based  scenario  analysis  and  non¬ 
attrition  based  effectiveness  measures  are  all  factors  to 
be  accommodated  in  future  modeling  requirements. 

Review  of  likely  future  issues  provides  more 
weight  to  the  notion  that  flexibility  will  be  the  key  to 
future  effectiveness  and  trade-off  analyses. 
Programmatics  will  play  a  larger  role  than  ever  in 
acquisition  decision  issues.  Effectiveness  and 
programmatic  concerns  must  remain  separate  but 
complementaiy  processes  must  be  developed  to  aid  the 
decision  maker.  If  this  is  to  be  accomplished,  separate 
and  complementaiy,  then  focus  on  the  development  of 
separate  and  complementaiy  modeling  processes.  This 
is  a  simple  view,  but  start  simple  and  build  in  difficultly 
as  the  particular  decision  issue  warrants.  With 
programmatic  and  affordability  issues  uncertainly  lies 
in  the  area  of  budgets,  force  structure  levels  and 
plausible  scenarios  but  weapon  system  effectiveness 
issues  deal  with  service  doctrine  and  system 
performance  within  defined  forces  and  doctrine. 

We  have  done  little  more  than  review  the 
environment  of  future  modeling  and  it  appears  that 
several  traits  must  be  reflected  in  our  next  generation  of 
modeling.  Without  these  traits  it  seems  that  modeling 
will  poorly  serve  the  decision  makers.  If  we  understand 
the  impact  and  critical  nature  of  these  model 


traits/characteristics  we  will  be  much  more  informed 
about  the  appropriate  techniques  and  tools  to  apply. 
(Specific  tools  are  discussed  below.) 

The  subgroup  developed  the  following  three 

critical  characteristics  of  future  COEA  effectiveness 

models; 

(1)  COEA  modeling  support  will  have  to 
deal  effectively  with  more  than  one  service 
and  over  long  time  periods. 

(2)  COEA  modeling  must  be  sensitive  to 
the  decision  issues,  i.e.  tailored  to  system 
issues  and  decision  maker  issues  for  the 
specific  COEA.  Interoperability,  lethality, 
supportability,  deployability  and  sustainability 
may  be  equally  weighted  COEA  issues. 

(3)  A  mix  of  tools  and  models  will  likely 
be  used  to  produce  a  thorough  analysis, 
because  no  one  model  or  technique  can 
effectively  address  such  a  wide  range  of 
issues. 

There  exists  strong  evidence  that  this  approach 
to  COEA  related  modeling  works  because  of  three 
examples  of  past  COEAs.  In  the  LH,  KE-ASAT  and 
BLK-III  COEAs  the  study  director  used  unique 
modeling  tools  to  address  the  critical  issues  of  the 
COEA.  In  the  case  of  the  LH  and  BLK-III  a  unique, 
not  previously  used,  mathematical  model  was 
developed  specifically  for  the  study.  In  the  case  of  the 
KE-ASAT  COEA  a  unique  war  game  seminar  was 
conducted  to  serve  as  the  medium  for  concept 
development  and  COEA  issue  analysis.  These  COEA 
analytical  models  were  effective  in  dealing  with  the 
issues  of  their  specific  acquisition  process.  Their 
efforts  relied  heavily  on  data  derived  from  our  current 
set  of  high  and  low  resolution  simulations  but  the 
current  set  of  models  and  simulations  were  used  as 
support  elements  only.  In  each  case  data  from  current 
models  was  used  as  a  driver,  an  emulator  or  stimulator 
for  a  more  specialized  model  focusing  on  specific 
COEA  issues.  Using  our  current  set  of  models  and  data 
as  a  foundation,  or  backdrop,  for  a  new  and  more 
flexible  set  of  tools  is  an  evolutionary  approach  to 
dealing  with  this  difficult  problem. 

What  effectiveness  methodologies  serve  the 
analyst,  and  decision  maker,  given  the  changing 
conditions  and  diversified  issues  presented  above?  No 
one  approach  to  effectiveness  analysis  can  be  adopted 
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since  the  environment  is  too  uncertain  and  tomorrow's 
issues  are  just  too  varied,  essentially  undefined.  Don't 
draw  upon  the  last  two  decades  of  effectiveness 

analysis,  threat  based  and  attrition  dependent,  to 

develop  your  future  model  criteria.  Listed  below  are 
several  analytical  methods,  modeling  techniques,  which 
are  well  suited  to  the  decision  making  and  modeling 
environment  of  the  future; 

1.  AHP 

2.  Utility  theory 

3.  Mathematical  programming  (linear, 
goal,  non-linear) 

4.  War  gaming 

5.  Unique  mathematical  modeling 

6.  State  space  modeling  techniques 

The  first  four  techniques  are  well  documented 

and  provide  tremendous  flexibility  in  dealing  with 
effectiveness  issues.  Each  must  be  tailored  to  address 
specific  issues  and  this  is  the  task  of  the  analyst. 
Unique  mathematical  modeling  could  include  the 
elaborate  combat  simulation  models  we  have  grown 
accustomed  to  over  the  years,  but  the  trend  will  likely 
be  towards  more  specialized  math  models  similar  to  the 
LH  COEA  model.  It  is  important  to  note  that  our 
cuirent  models  serve  a  significant  role  in  all  of  the 
above  mentioned  techniques,  especially  the  high 
resolution  model.  The  aggregated  nature  of  most  of 
these  techniques  relies  heavily  on  a  fundamental 
understanding  of  the  dynamics  of  the  specific  analytical 
issue.  In  this  regard  the  current  high  resolution  model 
may  play  an  even  more  crucial  role  as  a  data  provider, 
driver,  emulator  or  stimulator  for  the  more  aggregated 
form  of  modeling.  There  appears  to  exist  a 
fundamental  requirement  for  high  resolution  modeling 
to  address  system  performance  concerns  for  the 
warfighter,  fightability  and  trainability,  but  the  next 
level  of  issues  seems  to  rest  in  very  aggregated 
perspectives  and  expected  performance  for  forces  over 
long  time  periods,  a  military  campaign.  The  challenge 
of  the  military  analyst  is  to  develop  modeling 
techniques  to  use  high  resolution  insights  within  a 
campaign. 

State  space  modeling  techniques,  a  general 
form  of  the  Markov  model,  present  another  powerful 
set  of  modeling  tools  that  has  seen  limited  use  in  DOD 
combat  modeling.  The  Air  Force  has  used  this 
technique  for  limited  theater  level  analysis  as  well  as  B- 
52G  upgrade  effectiveness  analysis  and  F-16  mission 
performance  analysis.  Flexibility  and  transparency  are 
two  traits  of  state  transition  modeling.  It  unlike  several 


other  techniques  can  deal  directly  with  uncertainty  by 
modeling  expectations  and  likelihood  of  units 
performance.  Considerable  work  has  been 
accomplished  at  the  Naval  Postgraduate  School  in 
applying  this  particular  modeling  methodology  to  a  host 
of  military  effectiveness  issues. 

CAMPAIGN  ANALYSIS 

The  ambiguity  of  campaign  analysis  creates 
difficulty  in  adopting  an  appropriate  modeling 
technique.  As  outlined  above  COEA  related  campaign 
analysis  will  be  tailored  to  COEA  requirements 
therefore  methodologies  used  to  satisfy  the  modeling 
requirement  will  likely  be  defined  by  the  COEA.  The 
above  discussion  focused  on  the  applications  of  models 
to  a  specific  weapon  system  COEA,  but  is  there  a  need 
for  a  broader  more  general  model  application  that 
transcends  a  specific  weapon  system  requirement  and 
functions  as  an  "omnibus"  model  for  force  effectiveness 
issues?  Although  the  subgroup  had  little  time  to 
discuss  this  issue  there  is  without  a  doubt  a  need  for 
some  form  of  omnibus  campaign  analysis  methodology. 
Interestingly  though,  the  rational  for  flexibility  and 
transparency  for  COEA  related  campaign  analysis 
should  hold  true  for  a  more  general  campaign  analysis 
tool  or  tools.  It  is  within  the  context  of  more  general 
force  effectiveness  issues  and  affordability  issues  that 
omnibus  campaign  analysis  adopts  a  distinctive 
character  from  that  of  COEA  related  campaign 
modeling.  Within  COEA  analyses  the  general  focus  of 
the  analysis  is  on  "which  system  is  better  suited"  for  the 
force.  The  service  has  decided  to  commit  to  the 
acquisition  and  has  conducted  analyses  to  justify  its 
application  within  their  mission.  However, 
affordability  and  omnibus  issues  generally  focus  on 
"how  much  is  enough."  So  long  as  there  is  potential  for 
services  to  compete  for  national  defense  missions  the 
"how  much  is  enough"  analysis  should  lie  outside  the 
purview  of  the  COEA  (see  Prof  Hughes'  comments). 
From  an  effectiveness  methodologies  viewpoint  these 
are  not  totally  independent  processes  but  don't 
presuppose  that  one  modeling  methodology  will 
satisfactory  address  each  concern. 

The  best  example  for  a  candidate  methodology 
for  omnibus  campaign  analysis  is  exhibited  in  the  KE- 
ASAT  COEA.  A  war  game  seminar  provides  a  robust 
and  dynamic  forum  for  the  investigation  of  concepts 
and  critical  issues  as  well  as  generation  of  meaningful 
quantitative  data. 
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The  following  section  is  a  commentary  by 
Professor  Wayne  Hughes  from  the  Naval  Postgraduate 
School.  Professor  Hughes  was  asked  by  Dr.  Peter 
Purdue,  a  subgroup  member,  for  his  thoughts  on 
campaign  analysis.  Prof  Hughes  is  very  familiar  with 
the  campaign  analysis  issues  and  military  modeling 

methodologies. 

Campaign  and  Theater  Level  Analysis  for  COEA 

"A  campaign  analysis  is  a  low  resolution, 
quantitative  examination  of  heterogeneous  forces 
operation  over  a  large  geographic  area  in  a  series  of 
engagements  over  time.  It  is  the  analytical  equivalent 
of  military  campaign  executed  under  an  operations  plan 
that  defines  a  series  of  operations  by  integrated  forces. 
Its  conduct  is  as  distinctive  from  a  detailed  systems 
analysis  of  the  new  system  as  the  conduct  of  a 
campaign  is  distinctive  from  the  handling  of  forces  in 
a  battle. 

In  general,  campaigns  may  be  studied  with 
mathematical  models,  computer  simulations,  and  war 
games.  Each  of  these  three  forms  has  its  advantages 
and  disadvantages,  depending  on  the  operation  or 
analytical  purpose.  In  general,  a  combination  of  all 
methods  is  superior,  so  that  each  form  complements  the 
others.  A  family  of  campaign  analyses,  each  of  which 
is  done  with  a  specific  aim  (which  aim  may  or  may  not 
have  a  procurement  orientation),  will  in  the  aggregate 
provide  a  foundation  for  studies  with  many  aims, 
among  which  is  that  of  a  COEA. 

A  campaign  analysis  is  at  the  apex  of  a 
hierarchy  of  engagement  analyses  of  weapon  and 
sensor  performance  and  command  and  control.  It  is  a 
synthesizing  structure  which  presupposes  analysis  in 
detail  and  considerable  prior  aggregation  of  results. 

Because  of  the  almost  limitless  possibilities 
and  variations  in  detail,  is  particularly  well  to  remember 
the  If-Then  nature  of  every  model.  A  campaign 
analysis  is  a  stylized  set  of  inputs  and  dynamic 
relationships  (the  model)  from  which  flow  results. 
Because  of  the  tenuous  grasp  of  detail  at  the  campaign 
level  in  advance  of  actual  military  operations, 
successful  and  utilitarian  campaign  models  to  assist 
with  procurement  decisions  have  tended  to  be  highly 
abstract,  flexible,  and  transparent,  with  reproducible 
results,  involving  many  alternative  situations  and  cases. 
Thus,  while  computer  simulations  and  war  games  serve 
many  purposes  in  general,  relatively  concise 


mathematical  expressions  of  aggregated  interactions  are 
usually  best  for  COEA  in  particular. 

Mathematical  structure  often  suggests  some 
form  of  optimization  algorithm  to  derive  a  "best" 
solution.  When  a  mathematical  optimization  is 

possible,  it  should  not  be  dismissed  out  of  hand.  But  it 
is  far  more  important  when  dealing  with  campaign 
analyses  to  emphasize  their  value  for  exploratory 
computations,  in  which  diverse  points  of  view  are 
examined  and  in  the  best  of  worlds  the  analysis  assists 
in  a  convergence  and  unity  of  perspective  regarding  the 
strategic  environment  and  the  place  where  candidate 
systems  performs  its  mission. 

Moreover,  insofar  as  a  COEA  is  concerned,  a 
single,  new,  entirely  original  campaign  analysis  is 
insufficient.  The  productive  role  of  campaign  analysis 
is  achieved  when  there  is  already  in  hand  a  reasonably 
well  understood  sequence  of  studies  and  analyses  of  the 
theater  in  question  for  existing  and  future  forces,  both 
enemy  and  friendly,  into  which  the  new  system  in 
question  may  be  inserted. 

Specifically  with  regard  to  the  Phases  of  the 
Acquisition  Process,  the  following  comments  apply: 

1.  Preliminary  to  Phase  0:  The  acquisition 
process  envisions  the  determination  of  mission  need.  If 
the  need  is  to  fulfill  a  new  mission,  as  is  commonly  the 
situation  that  has  existed  in  the  armored  forces  since 
around  1989,  then  only  a  campaign  or  theater 
framework  can  serve  to  test  the  efficacy  of  the  new 
mission,  the  candidate  weapon,  sensor  or  other  system 
to  fulfill  it,  and  the  possible  alternatives. 

If,  for  example,  the  umbrella  mission  is  to 
safeguard  the  movement  of  goods,  military  or 
commercial,  at  sea,  a  prominent  sub-mission  has  been 
to  protect  against  submarines.  The  many  questions 
about  the  nature  of  the  present  submarine  threat 
illustrate  the  most  basic  Pre-Phase  0  issue.  Given  that 
campaign  analysis  has  narrowed  these  diverse  points  of 
view  and  the  future  threat  variations  are  sufficiently 
agreed,  then  the  issue  becomes  the  strategy  to  defeat 
submarines.  Until  recently,  a  long  series  of  campaign 
analyses,  other  more  detailed  studies,  and  many  at  sea 
exercises  had  evolved  an  "offensive  strategy,"  which  is 
to  say  to  new  submarines  threat  characteristics  may 
upset  that  strategy.  The  degree  of  upset  and  need,  if 
any,  to  devise  a  new  strategy  is  almost  exclusively  in 
the  campaign  analysis  domain. 
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It  is  possible  that  the  need  for  a  new  system  or 
technolog}'  to  repair  a  bankrupt  strategy  may  be 
uncovered  in  this  way.  It  is  more  common  that  the  need 
will  be  adumbrated— this  is,  sketched  out  and 
publicized— by  campaign  analysis.  This  arises  the 
consciousness  of  new  missions,  or  a  new  strategy 
within  a  mission,  Usually  the  adumbration  is 
accompanied  by  a  candidate  solution,  as  for  instance,  a 
new  ASW  submarine  or  a  new  multipurpose  submarine 
with  several  mission  capabilities. 

The  thrust  of  these  comments  are  to  emphasize 
that  anything  more  than  a  superficial  and  unconvincing 
understanding  of  new  mission  or  strategic  requirements 
come  from  a  strong  foundation  of  continuing  campaign 
studies  at  the  theater,  that  is,  strategic  and  operational, 
levels  of  thought. 

2.  Phase  0:  Concept  Exploration  and 
Definition:  For  most  acquisitions  a  new  version  of  an 
old  system  is  proposed  to  perform  a  longstanding 
mission.  The  new  one  is  justified  on  the  basis  of  (a)  a 
threat  upgrade,  (b)  a  new  technology  breakthrough,  (c) 
a  replacement  as  preferable  to  a  SLEP,  or  (d)  a 
combination  of  the  three.  At  such  a  juncture,  it  is 
reasonable  to  ask  whether  an  entirely  different  system 
might  be  superior.  For  instance,  if  a  replacement 
submarine  class  is  proposed  for  ASW,  a  fair  question  is 
whether  an  ASW  aircraft,  mines,  or  even  Air  Force 
aircraft  might  be  preferable.  A  campaign  analysis  is  the 
preliminary  methodology  to  describe  the  way  the 
alternative  system  would  fulfill  the  mission. 

But  such  a  campaign  analysis  can  do  no  more 
than  establish  the  feasibility  of  the  alternative.  If  it  is 
thought  that  some  type  of  mines  could  substitute  for  a 
new  submarine  class,  then  a  vast  amount  of  analysis 
must  follow.  In  addition,  it  is  not  practical  to  assign  the 
burden  of  proof  to  the  submarine  advocate  that  the 
submarine  is  preferable  to  any  and  all  alternatives.  For 
one  thing,  this  leads  to  an  endless  series  of  explorations, 
rather  like  a  judge  demanding  the  proof  of  innocence  of 
the  accused.  For  another,  the  submarine  advocate  will 
not  likely  grasp  or  express  the  advantages  of  the 
competing  system.  A  proposed  replacement  in  kind 
may  be  justified  on  the  basis  of  threat,  technology,  and 
cost,  but  the  burden  of  demonstrating  (to  some 
preliminary  degree)  the  superiority  of  an  alternative 
system  such  as  a  mine  lies  with  the  proponent  of  the 
alternative.  The  advocacy  of  competing  systems  may 
be  carried  out  by  the  advocates,  but  the  campaign 
analysis  of  mission  needs  entails  a  neutral,  objective 
team. 


3.  Phase  0  Camnaign/Iheater  Level  Analysis: 
Let  us  postulate  that  there  is  the  required  foundation  of 
existing  campaign  studies  of,  for  example,  the 
protection  of  shipping  and  military  forces  at  sea  which 
includes  the  role  of  ASW  submarines.  Let  us  further 
postulate  that  we  allow  the  advocate  of  a  new  ASW 
submarine  class  to  proceed  with  concept  studies.  The 
constructive  role  of  campaign  analysis  is  at  that 
juncture  highly  efficacious.  The  "approved"  settings 
(or  scenarios)  exist,  as  do  the  base  of  existing  friendly 
forces,  their  capabilities  and  their  employments.  The 
threat,  including  future  trends,  is  in  hand. 

The  primary  purpose  of  the  COEA-specific 
campaign  analysis  is  to  place  the  new  submarine 
capabilities  in  operational  context  with  one  or  more 
intelligent  employment  variations.  If  the  change  in 
submarine  capabilities  is  radical,  other  force 
employments  may  have  to  be  rearranged,  but  that  is  an 
attainable  goal.  A  utilitarian  campaign  analysis  will 
elucidate  the  proper  measure  of  force  effectiveness 
(MOFE). 

It  is  not  likely,  however,  that  the  cost- 
effectiveness  of  the  new  system  can  be  deduced  from  a 
campaign  study.  The  analysis  is  too  coarse-grained 
(low  resolution)  to  infer  system  performance  on  the 
margin.  Often  a  simpler,  straightforward,  proxy 
measure  of  effectiveness  (MOE)  can  be  deduced  and 
agreed.  For  example,  it  is  usually  accepted  that 
safeguarding  the  arrival  of  cargo  is  the  true  aim  of 
ASW  forces,  and  that  two  robust  MOFEs  of  the  ASW 
campaign  are  percentage  of  ships  lost  (for  early  and 
critical  cargo)  and  exchange  ratio  of  ships  lost  per 
submarine  sunk  (for  an  extended  campaign). 
Conjecture  that  a  new  ASW  submarine's  campaign 
employment  kills,  but  does  not  deter  or  suppress, 
enemy  submarines.  If  so,  we  know  from  the  campaign 
analysis  that  a  suitable  proxy  MOE  for  the  ASW 
submarine  is  the  probability  that  a  transiting  submarine 
is  killed  under  scenario  conditions  specified  by 
the  campaign  setting.  The  campaign  analysis  also 
established  the  value,  vis-a-vis  the  MOFE,  of  every 
level  of  kill  (MOE)  attainment.  The  campaign  analysis 
describes  the  types  of  enemy  submarines  and  the 
proportion  of  each  to  be  engaged  by  the  ASE 
submarine.  Finally,  the  campaign  analysis  will 
probably  reveal  the  extent  to  which  the  sub-on-sub 
exchange  rate  is  a  decisive  factor  in  the  analysis. 

In  this  way  the  COEA-related  campaign 
analysis  defines  the  setting  for  a  subsequent  system 
analysis,  which  must  be  conducted  in  great  detail. 
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including  tactics  by  which  the  enemy  is  detected, 
classified,  closed,  engaged,  and  killed,  and  the 
associated  probabilities  for  each  enemy  type  variant. 
The  detailed  cost  and  effectiveness  studies-the  systems 
analysis  proper-can  then,  and  only  then,  proceed  with 

confidence  that  kill  probability  is  an  adequate  MOE  and 
that  the  specific  systems  analysis  setting  (scenario)  is 
robust  and  well-defined.  Finally,  if  the  analysis 
suggests  that  :  ub-on-sub  exchange  ratio  is  going  to 

be  unexpected  gh,  then  the  campaign  analysis  will 
have  revealed  aether  or  not  this  is  a  problem  that 
requires  a  review  of  the  MOE,  and  indeed  the  entire 
strategy. 


In  conclusion,  a  whole  series  of  past  campaign 
analyses  is  a  necessary  antecedent  to  serve  as  the 
foundation  for  a  specific,  simple,  focused,  reproducible 
campaign  analysis  to  support  the  COEA.  The  focused 
campaign  analysis  is  a  necessary  antecedent  of  the 

detailed  systems  analysis  of  the  system  at  issue,  to 
determine  its  cost-effectiveness.  The  general  campaign 
study  foundation,  the  COEA-specific  campaign 
analysis,  and  detailed  cost-benefit  systems  analysis  are 
three  distinguishable  efforts  that  cannot  be  combined." 
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SETTING  UP  THE  ANALYSIS 

PROCESS  ISSUES 


POOR  COMMUNICATIONS  BETWEEN  CONSUMERS 
AND  PRODUCERS 

•  ANALYSIS  TEAM  FAR  REMOVED  FROM  DECISION 

MAKERS 

•  NO  DIRECT  PATH 

•  ENVIRONMENT  CHANGES  OVER  DURATION  OF 
ANALYSIS 

NOT  STRUCTURED  TO  LEARN  LESSONS  FROM  OTHER 
COEAs 

•  SUCCESSFUL  METHODOLOGIES  AND  MEASURES 

•  EXAMPLE  ROADMAPS 

•  FEEDBACK  ON  PERFORMANCE 

DIFFICULT  TO  MAINTAIN  CONTINUITY  OVER  DURATION 
OF  COEA 

•  ENVIRONMENT  CHANGES 

•  PLAYERS  CHANGE 

•  POLICY/GUIDANCE  EVOLVES 


IMPROVING  THE  PROCESS  WILL  IMPROVE  THE  PRODUCT 


SETTING  UP  THE  ANALYSIS 

RESEARCH  ISSUES 


DIFFICULT  TO  STRUCTUE  APPROACH  DECISION 

MAKERS  CAN  USE 

•  DECISION  MAKERS  NOT  COMFORTABLE  WITH 
OR  NUANCES 

•  ANALYSTS  DO  NOT  UNDERSTAND  DECISION  MAKERS 
THINKING 

•  HOW  TO  RELATE  ANALYSIS  TO  OTHER  DECISION 
CRITERIA 

INSUFFICIENT  CRITERIA  TO  SELECT  APPROPRIATE 

MEASURES 

•  METRICS  WITH  MEANING  IN  TODAY’S  DECISION 
ENVIRONMENT 

•  METRICS  FOR  EVOLVING  WARFARE  NOT  DEFINED 

•  HOW  TO  RELATE  CAMPAIGN  METRICS  WITH  MOEs 
AND  MOPs 

NO  CONSENSUS  ON  QUALITATIVE  MEASURES 

•  MOST  MEASURES  APPLY  TO  WEAPONS  SYSTEMS 

•  MEASURES  FOR  INFORMATION/DECISION  SYSTEMS 
NOT  DEFINED 


OR  COMMUNITY  NEEDS  TO  DEVELOP  NEW  TOOLS 
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SETTING  UP  THE  ANALYSIS 

KEY  RECOMMENDATIONS 


•CREATE  A  PROGRAM  TO  ADDRESS  ONGOING  COEA  ISSUES 

-MAINTAIN  A  LIBRARY  OF  COEA  LESSONS  LEARNED 
-CONDUCT  “OUTREACH”  ACTIVITIES 
-DEVELOP  NETWORK  OF  SPECIALISTS 

•RESEARCH  TO  DEVELOP  INNOVATIVE  DECISION  MAKING  SUPPORT 

-INFORMATION  DISPLAY 

-INSIGHT  INTO  ANALYTIC  PROCESS  AND  CONTENT 
-DECISION  MAKER  INTERACTION 

•DEFINE  METRICS  FOR  EMERGING  CONFLICT  ENVIRONMENTS 

-JOINT/COMBINED  OPERATIONS 
-NON-WEAPON  SYSTEMS 

-CONTINGENCY  AND  LIMITED  OBJECTIVE  WARFARE 


APPLY  MODERN  TECHNOLOGY  TO  IMPROVE  THE  PROCESS 


EFFECTIVENESS  METHODOLOGIES 
COMPARATIVE  MODELS 


OBJECTIVE: 


•TO  IDENTIFY: 

-  ISSUES/PROBLEMS 

-  RESEARCH  AREAS 

-  FUTURE  DIRECTIONS 


THAT  APPLY  TO  THE  EFFECTIVENESS 
PORTION  OF  THE  COEA  PROCESS 


EFFECTIVENESS  MODELS 
PROBLEMS/ISSUES 


SHORTCOMING  -  CHARACTERIZATION  OF 
NON-QUANTITATIVE  EFFECTS 

MANY  CURRENT  MODELS  ARE  NOT  ROBUST 
ENOUGH  TO  HANDLE  THE  CHANGING 
THREAT  ENVIRONMENT 

USE  OF  CONTRACTOR  PROPRIETARY 
MODELS 

TRADE  OFF  BETWEEN  EFFICIENCY 
AND  FIDELITY 


EFFECTIVENESS  MODELS 

RESEARCH  AREAS 


MODEL  EVALUATION  CRITERIA 

DEFINITION  AND  ACCEPTABILITY 
OF  MODEL  CATEGORIES/HIERARCHY 
FOR  USE  IN  COEAs 

VARIABLE  RESOLUTION  METHODOLOGY 
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EFFECTIVENESS  MODELS 
FUTURE  DIRECTION 


USE  OF  WARGAMING  IN  COEA  PROCESS 
JOINT  ANALYSIS 

DISTRIBUTED  INTERACTIVE  SIMULATIONS 

FLEXIBLE/MODULAR  MODELS 

-  RAPID  PROTOTYPING 

-  DATA  DRIVEN 


CAMPAIGN  ANALYSIS 


•  NEED  GUIDANCE  TO  THE  SERVICES 

•  HARD  TO  DEFINE  THE  PROBLEM 

•  SCOPE:  “LIMIT  TO  COEA  USE” 
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CAMPAIGN  ANALYSIS 

METHODOLOGIES 


•  BUILD  UPON  CURRENT  INVENTORY 

•  UNIQUE  MATHEMATICAL  MODELS  (LH  COEA) 

•  WARGAMING  SEMINARS  (KE-ASAT) 

•  LIBRARY  TYPE  DATA  SUPPORT 

(OBJECT  ORIENTED  THINKING) 

•  STATE  SPACE  MODELING 


CAMPAIGN  ANALYSIS 
(KE-ASAT  EXAMPLE) 

CAMPAIGN 

ANALYSIS 

OPERATIONAL 

ANALYSIS 

SYSTEM 

ANALYSIS 

MS  1  EEAs/MOEs 
Alter  concepts 
Threat  defined 
Reqmnt  Trade-offs 
Scenarios 
Military  value 

MS  II  Refine  MS  1 

Operational 

value 

of 

system 

defined 

same  as  MS  1 

Compare 

concepts 

Sense  Anal 

same  as  MS  1 

TOOLS 

Analytical  Wargame 

Force  level 

System  level 

Model 

Model 

CAMPAIGN  ANALYSIS 

METHODOLOGIES 


•  BUILD  UPON  CURRENT  INVENTORY 

•  UNIQUE  MATHEMATICAL  MODELS  (LH  COEA) 

•  WARGAMING  SEMINARS  (KE-ASAT) 

•  LIBRARY  TYPE  DATA  SUPPORT 

(OBJECT  ORIENTED  THINKING) 

•  STATE  SPACE  MODELING 


CAMPAIGN  ANALYSIS 

(KE-ASAT  EXAMPLE) 


CAMPAIGN 

ANALYSIS 

OPERATIONAL 

ANALYSIS 

SYSTEM 

ANALYSIS 

EEAs/MOEs 

Operational 

Compare 

Alter  concepts 

value 

concepts 

Threat  defined 

of 

Sense  Anal 

Reqmnt  Trade-offs 

system 

Scenarios 

defined 

Military  value 

Refine  MS  1 

same  as  MS  1 

same  as  MS  1 

MS  II  f 


TOOLS 


Analytical  Wargame  Force  level 

Model 


System  level 
Model 
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4.5  Report  of  the  Costing  Methodologies 
Working  Group 

We  had  vigorous  discussion  by  45  participants 
who  represented  OSD,  Major  Commands,  Laboratories, 
the  academic  community  and  hardware  vendors.  The 
success  of  the  working  group  depended  on  all  of  them, 
with  special  thanks  to  LtCol  Ray  Baker  of  the  Navy 
War  College  for  administrative  and  logistical  support 
and  to  Don  Mixon  of  the  MITRE  Corporation  for 
organizational  support. 

Everyone  agreed  that  special  emphasis  should 
be  placed  on  the  fact  that  the  first  letter  in  COEA  is  a  C, 
and  that  it  stands  for  COST.  Without  proper  and  early 
attention  to  cost,  the  COEA  process  is  fatally  flawed. 

The  rest  of  this  report  is  keyed  to  the  attached 
briefing  slide. 

Service  Cost  Estimate  Linkage 

There  had  been  a  concern  among  the 
participants  in  the  working  group  over  whether  each 
Service  has  a  process  to  support  the  various  cost 
estimates  that  have  to  be  prepared  for  program 
manager's  estimates,  COEAs,  and  independent  cost 
estimates  (ICEs).  We  decided  that  there  is,  within  each 
service  a  process  and/or  organization  which  prepares 
these  various  cost  estimates.  Further,  as  the  chart 
shows,  the  processes  and  organizations  differ  from  one 
service  to  the  next,  but  that  the  lack  of  organizational 
uniformity  is  not  worrisome. 

Consistency  In  Cost  Estimates 

The  new  DODI  5000  series  requires  that  there 
be  consistency  among  the  various  cost  estimates  that 
have  to  be  prepared  for  program  manager's  estimates, 
COEAs,  and  independent  cost  estimates  (ICEs).  There 
was  concern  that  there  were  not  processes  in  place 
within  the  Services  to  ensure  that  consistency.  After 
discussing  the  issue  within  the  Working  Group,  we 
decided  that  there  is,  for  each  service,  a  consistency 
filter. 

•  For  the  Army,  USACEAC  participates  in  the 
creation  of  the  Army  Cost  Position,  so  it  is  in  the 
position  to  ensure  consistency  among  the  various  cost 
estimates. 

•  The  Air  Force  holds  its  own  internal  CAIG, 
and  this  forum  permits  enough  insight  into  the  various 


cost  estimates  to  provide  the  requisite  consistency 
check. 

•  For  the  Navy,  the  Naval  Center  for  Cost 
Analysis  (NCA)  generates  the  ICE,  reviews  the  PM's 
estimate,  and  serves  on  the  COEA  Oversight  Board. 
This  "triple-threat"  by  the  NCA  provides  the  Navy  with 
a  consistency  filter.  [Now  I  get  to  tell  my  boss,  the 
Director  of  the  Naval  Center  for  Cost  Analysis,  of  this 
newly-discovered  mission  of  the  organization.] 

Cost  Approaches  In  Support  Of  Acquisition  Phases 

We  discussed  the  cost  analysis  methodologies 
that  exist  and  their  use  in  the  various  milestone  phases. 
As  a  program  moves  from  Milestone  I  to  II  to  III,  the 
specificity  of  program  definition  increases  (and 
uncertainty  decreases),  but  the  uncertainty  is  embedded 
in  the  programmatics,  rather  than  in  the  CERs  that  are 
used  for  the  cost  estimate.  We  concluded  that  the 
advent  of  COEAs  does  not  alter  this  matrix  or  this 
observation.  We  also  observed  that  the  validity  of 
CERs  is  like  cheese  —  it  has  a  perishable  shelf  life;  As 
the  underlying  data  base  ages  and  obsolesces,  so  does 
the  validity  of  the  CER  decline.  There  is  a  constant 
need  to  update  data  bases  in  order  to  reflect  current 
technology  and  acquisition  environments. 

If,  for  example,  DoD  enacts  an  acquisition 
policy  is  enacted  that  fosters  development  of 
technologies  but  forgoes  their  incorporation  into 
production  units,  then  R&D  costs  will,  arguably,  rise 
significantly,  and  R&D  CERs  will  have  to  be  updated 
to  reflect  this  new  acquisition  environment. 

Source  Of  Estimating  Risk  And  Uncertainty 

Risk  and  uncertainty  are  major  issues  in  cost 
estimation.  Of  the  four  sources  of  estimating  risk  and 
uncertainty,  three  of  them  (uncertainty  in  requirements, 
uncertainty  in  the  ultimate  design  solutions  and 
uncertainty  in  the  timely  availability  of  appropriate 
technology)  are  program-related,  while  one  (the 
inherent  uncertainty  in  statistically  based  estimating 
methods)  has  to  do  with  the  adequacy  of  cost  estimating 
tools.  The  working  group  concluded  that  the  cost 
analysis  community  has  to  be  involved  in  the  analysis 
process  from  the  earliest  possible  time,  and  its 
involvement  should  continue  throughout  the  analysis 
process. 
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Summary 

The  items  on  this  chart  summarize  the 
important  items  from  the  working  group. 

•  Linkage  and  consistency  among  cost  estimates 

is  important,  and  seems  to  be  in  hand. 

•  Risk  and  uncertainty  cannot  be  ignored. 
Program  risk  should  abate  as  a  program  moves  through 
its  milestone  process. 


•  Updating  cost  tools  is  a  continuing  process, 
which  requires  people  and  funds.  Cost  estimating  tools 
are  perishable  as  the  acquisition  environment  (e.g., 
business  base,  acquisition  strategy,  etc.)  changes. 

•  Early  involvement  by  the  cost  estimating 

community  is  critical  for  credible  and  useful  COEAs. 
Cost  estimating  is  important  and  adds  value  to  the 
acquisition  process.  In  fact,  the  cost  estimating  process 
enhances  understanding  of  the  program  by  forcing 
greater  clarity  in  program  definition. 
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MORS  COEA 
MINI-SYMPOSIUM 
MARCH  9-11,  1992 


COSTING  METHODOLOGY 
WORKING  GROUP 


CHAIRMAN 
Dr.  Dan  Nussbaum 
NCCA 


THE  FIRST  WORD  IN  COEA  IS... 


COST 


SERVICE  COST  ESTIMATE  LINKAGE 


ARMY 

PM 

COEA 

ICE 

ACP 

NAVY 

PEO/AMC 

TRADOC 

CEAC 

CEAC 

(PM/COEA/ 

SYSTEM 

COMMAND 

SYSTEM 

COMMAND 

NCA 

ICE) 

USAF 

PRODUCT 

DIVISION 

OPERATIONAL 

COMMAND 

PRODUCT 

DIVISION/ 

SAF(FM) 

BOTTOM  LINE:  THREE  DIFFERENT  PROCESSES 
ALL  SEEM  TO  WORK 


CONSISTENCY  IN  COST  ESTIMATES 


HOW  DO  WE  ENSURE  CONSISTENCY  BETWEEN 
THE  COEA  AND  SERVICE  COST  ESTIMATES 
PREPARED  FOR  THE  DAB? 

ENSURED  BY  SERVICE  COST  REVIEW 

-  ARMY  COST  POSITION 

-  NAVY  COEA  OVERSIGHT  BOARD 

-  USAF  CAIG  REVIEW 
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COST  APPROACHES  IN  SUPPORT  OF 
ACQUISITION  PHASES 


MILESTONE  O  I  II  III 


COST TOOL 

PARAMETRIC 

D 

X 

B 

ANALOGY 

B 

X 

B 

BOTTOM  UP 

B 

X 

X 

X 

TIME  -  -  ►  DEFINITION  INCREASES 

UNCERTAINTY  DECREASES 


-  -►  TOOLS/CERS  NEED  UPDATING  AND  IMPROVEMENT 

CHALLENGE:  ADJUST  HISTORICAL  DATA  TO  REFLECT 

FUTURE  TECHNOLOGY  AND  ENVIRONMENT 
(THIS  COSTS  $$) 

EXAMPLE:  R&D  WILL  INCREASE  “SIGNIFICANTLY"  IF 

POLICY  TO  PUT  SYSTEMS  ON  THE  SHELF 
PREVAILS 


SOURCES  OF  ESTIMATING 
UNCERTAIN  RISK 


•  REQUIREMENTS  (THREAT) 

•  DESIGN  SOLUTIONS 

•  TECHNOLOGY  AVAILABILITY 


ESTIMATING  METHODS 


SOURCES  OF  ESTIMATING  &  ANALYSIS 
OF  ALTERNATIVES 
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4.6  Report  of  the  COEA  Management  Working 

Group 

Issue  #1:  Role  of  Contractor  in  the  COEA 

•  Similar  to  that  of  current  program  support  or 
SETA  efforts 

•  Impacts  are: 

1)  ability  to  provide  professional,  skilled 

resource  pool-continuity, 

2)  availability  of  unique,  specialized  analytical 
tools,  and 

3)  direct  cost  to  service/program  office 

•  Use  of  contractor  requires  careful  screening 

for  both  capability  and  ensuring 

independent/objective  work  (contractor 
accepting  such  work  must  be  willing  to 
exclude  the  company/corporation  from  other 
work  on  the  program) 

•  Independent/objective  work  is  not  likely  if 

contractor  "answers"  to  PM.  Best 

management  technique  is  to  have  contractor 
responsive  to  COEA  oversight  committee,  or 
similar  group. 

•  OSD  could  provide  list  of  desirable  skills  to 
help  in  preparation  of  RFP  and  contractor 
selection 

•  Draft  RFP  requirement  statements  may  help 

•  Team  stopped  short  of  recommending  OSD 

list  of  "approved"  or  "recommended" 

contractors  for  COEA  support  services 

Issue  #la:  Role  of  FFRDC  in  COEA 

•  FFRDCs  best  suited  for  general  COEA 
support,  not  work  related  directly  to  a  specific 
COEA. 

•  Rationale:  FFRDCs  usually  not  responsive  to 
hard  deadlines,  not  well-suited  to  represent 
service  interests  in  COEA.  FFRDC  talent  is 
better  suited  to  study  more  general  subject 
related  to  COEA  such  as  policy  implications, 
the  development  or  analysis  of  COEAs 


models,  the  development  or  analysis  of  COEA 
models,  the  development  of  new  analysis 
techniques,  etc. 

•  OSD  may  recommend  services  establish 
policies  consistent  with  the  FFRDCs  role  in 
COEA 

•  COEA  model  development  COEA  policy 
analysis  (Research  areas) 

•  Action:  MORS  recommend  process 

improvement  and  modeling  and  analytical 
techniques 

Issue  #2:  Structure  of  COEA  Oversight  Board 

•  Establish  with  experienced/senior  individuals 
from  key  involved  organizations  both  user  and 
acquisition  who  can  contribute  to  successful 
conduct  of  the  COEA. 

•  This  oversight  board  will  enhance  fhe 
credibility  of  the  analysis. 

•  Allows  for  a  good,  viable  method  to  include 
sister  service  input. 

•  Builds  consensus  support  for  the  COEA. 

•  Provides  authority  for  the  study  director. 

•  Core  members  of  oversight  body: 
user/combat  developer,  acquisition 
executive/system  developer,  intel  rep.,  T&E 
rep..  Materiel  developer/sy stems  command 
rep.,  cost  estimating  rep.  Departmental 
resource  allocation  manager,  i.e.,  AF/PE,  DA 
(PA&E),  OP-80  for  affordability  expertise), 
other  service  representatives,  OSD  (observer). 

Issue  #3:  Role  of  COEA  Oversight  Board 

•  Define  and  focus  the  COEA  on  key 
decisionmaker  issues. 

•  Recommend  approval  of  the  COEA  study 
plan. 

•  Assist  in  definition  of  COEA  issues. 
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Conduct  periodic  reviews  of  COEA  progress. 


•  Recommend  approval  and  release  of  COEA 
findings. 

•  Provide  advice  and  council  to  study  direction 

in  planning  and  executing  COEA. 

Issue  #4:  Support  and/or  Assistance  to  the  Study 
Director 

•  Establish  an  Advisory  Working  Group  (AWG) 
chaired  by  the  study  director. 

•  AWG  represented  by  each  organization  on  the 
oversight  board  and  other  COEA  participants. 

•  AWG  assists  in  the  formulation  of  the  COEA 
study  plan. 

•  AWG  members  assist  COEA  execution. 

Issue  #5:  Coordination  of  Contributors  (threat, 

data,  modelling,  cost,  tests,  and  others) 

•  In  reality,  this  is  not  an  OR  issue,  but  instead 
is  a  management  issue.  As  in  the  case  of  any 
effort  involving  a  variety  of  talents  and 
organization,  the  smart  move  is  to  coordinate 
efforts  via  a  plan.  This  plan  contains  elements 
pertaining  to  schedule,  tasking,  requiring 
resources  and  expected  output(s). 

•  Inherent  in  formulation  of  the  plan  are  inputs 
from  the  subordinate  organization  that  will 
support  production  of  the  final  product.  This 
insures  that  supporting  organizations  have 
some  sense  of  having  bought  into  the  plan. 

•  As  a  final  element,  the  plan  must  be  approved 
at  an  appropriate  level  of  seniority  such  that 
the  approved  plan  is  effective. 

Issue  #6:  Responsiveness  of  COEA  to  the  Changes 

•  Impact:  By  not  designing  the  initial  COEA  to 
be  responsive  to  future  changes,  the  next 
(follow-on)  COEA  may  be  just  as  hard  and 
resource  intensive  as  the  original  study;  and 
we  may  have  to  restart  at  MS  0,  or  even  could 
jeopardize  the  entire  program. 


•  Action:  Structure  one's  initial  COEA  broad 
enough  via  parametric  bounding  to  look  at 
"ranges"  of  threats,  scenarios,  effectiveness 
and  cost. 

Issue  #7:  Contribution  of  Senior  Service  Colleges/ 

Universities  Through  Enhancement  of  Education 

•  Impact:  Obviously,  senior 

colleges/universities  can  have  little  role  in 
performing  COEA,  due  to  timeliness, 
schedule,  and  accountability  issues.  However, 
they  can  contribute  immeasurably  to  educating 
senior  folks  as  to  COEA  processes  and  issues. 

•  Policy  implication:  No  direct  impact  upon 
policy. 

•  Research  Applications:  Primary  issue  would 
be  general  PME  regarding  COEA  processes 
and  issues.  Might  also  become  involved  in 
tool/process  building. 

•  Action:  Include  COEA  case  studies  in  PME. 
Examine  tool  applicability  including  but  not 
limited  to  wargaming  and  simulations. 

Issue  #8:  Focus  on  the  Service  Role  in  COEA 

Development 

•  PM  involvement  becomes  more  focused  past 
MSI. 

•  Essential  that  developer  be  involved  in  pre  MS 
I  analysis. 

•  Impact:  Service  is  best  qualified  in  operations 
environment  and  links  requirements  and 
programming  functions.  PA&E  interface. 

•  Problem  areas  are:  service  only  perspective 
and  questions  on  objectivity. 

•  Full  integration  of  joint  programs  and  theater 
level  analysis. 

•  Oversight  and  guidance:  Policy  implication  of 
senior  lead,  who  funds  effort  and 
accomplishes  analysis,  differences  in  service 
procedures,  who  decides  what  is  correct. 
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Research  Areas:  Review  different  processes 
for  COEA  effectiveness,  need  for  joint 


modelling,  development  of  campaign  analysis 
at  theater  level. 

•  Actions:  Examine  different  processes, 

development  of  flexible  tools,  can  the  process 
be  streamlined,  joint  service  modelling. 

•  Assumptions:  PM  (Developer)  has  to  be 
involved  throughout  the  process  with  role 
increasing  as  program  matures. 

•  Contractor  support  is  dependent  on  value 
added  capability. 

•  Warfighters  are  ultimately  responsible  for 
efforts. 

Issue  #8a:  Role  of  the  Program  Manager  in  COEA 

•  Impact:  PM  participation  in  COEA  is 
critically  important  but  participation  must  be 
balanced  by  equally  critical  participation  of 
"user"  or  "warfighting"  service 
representatives.  Role  of  PM  is  service 
dependent  and  also  dependent  on  program 
maturity  {PM  role  changes  as  program 
matures}.  PM  is  best  source  for  system 
capabilities,  realistic  program  schedules, 
developmental  costs,  etc.  PM  influence  may 
need  to  be  controlled  in  earliest  stages  of 
program  development  to  permit  objective 
consideration  of  widest  range  of  options 

•  Policy:  PM  role  is  service  controlled.  Service- 
run,  COEA  oversight  committees  are  probably 
best  suited  to  make  the  best  use  of  PMs 
contribution. 

•  Research  Area:  Unknown 


Issue  #9:  Potential  Management  Techniques  at  the 
Study  Director  Level 

•  Must  be  tailored  to  issue/program. 

•  Capability  required. 

•  Time/Resources  available. 

•  Study  director  (SD)  has  latitude  to  select. 

•  Cost  may  direct  size/scope  available. 

•  SDs  basic  management/leadership  style. 

Issue  #10:  Funding  and/or  Sponsorship  of  COEA 

•  MS  0/1,  service  HQs.  (see  viewgraph) 

•  Remember  reasons  for  new  capability,  to 
correct  a  deficiency,  and  exploit  an 
opportunity. 

•  Resources  for  studies  at  risk. 

Issue  #11:  Best  Organizational  Structure  to 

Facilitate  COEA  Processes 

•  See  viewgraph  for  organizational  structure 
answer. 

Closing  Statement 

Active  participation  of  both  user  and 
acquisition  communities  is  essential  to  successful 
completion  of  a  COEA! 


•  Action:  Use  PM  wisely. 
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MORS  COEA 
MINI-SYMPOSIUM 
MARCH  9-11,1992 


COEA  MANAGEMENT 

WORKING  GROUP 


CHAIRMAN 
LtCol  Don  Bourdon 
NAVAL  WAR  COLLEGE 


COEA  MANAGEMENT 


OVERSIGHT  BOARD  CORE  MEMBERSHIP 

•  WARFIGHTER/USER 

•  ACQUISITION/DEVELOPER 

•  TEST  AND  EVALUATION 

•  INTELLIGENCE  COMMUNITY 

•  COST  COMMUNITY 

•  PPBS/PROGRAMMING 
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COEA  MANAGEMENT 


DECISION  MAKER  ISSUES  THAT  MUST  BE 
ADDRESSED  BY  A  COEA 


WHY  IS  A  NEW  START  ACQUISITION 
REQUIRED  (ANALYTICAL  UNDERPINNINGS)? 

WHAT  DRIVES  EFFECTIVENESS? 

WHAT  DRIVES  COST? 

WHAT  DRIVES  THE  ACQUISITION  MILESTONE 
DECISION  SCHEDULE? 


COEA  MANAGEMENT 


FUNDAMENTAL  ISSUES  IN  COEA 
MANAGEMENT  (CONTINUED) 


DEVELOPING  SIMULATION  MODELS/TOOLS 
TO  PERMIT  TIMELY,  COST  EFFECTIVE, 
CREDIBLE  ANALYSIS  OF  JOINT  CAMPAIGN 
LEVEL  ISSUES  IS  A  PRIORITY  NEED 

BUILDING  A  TRUSTING  RELATIONSHIP 
BETWEEN  THE  WARFIGHTER,  THE 
ACQUISITION  COMMAND,  AND  OSD 
NECESSITATES  FULL  AND  OPEN  DISCUSSION 
AS  COEA  PROCEEDS 
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COEA  MANAGEMENT 


FUNDAMENTAL  ISSUES  IN  COEA 
MANAGEMENT  (CONTINUED) 


ESTABLISHING  AN  OVERSIGHT  BOARD/ 
SENIOR  ADVISORY  GROUP  TO  PROVIDE 
ADVICE  AND  COUNSEL  TO  STUDY  DIRECTOR 
IS  VITAL 

ESTABLISING  A  WORKING  LEVEL  STUDY 
TEAM/ADVISORY  GROUP  TO  SUPPORT  THE 
STUDY  DIRECTOR  IN  THE  EXECUTION  OF 
THE  COEA  FACILITATES  TIMELY  STUDY 
COMPLETION 


COEA  MANAGEMENT 


•ISSUE:  WHAT  SHOULD  BE  THE  ROLE  OF  THE  OVERSIGHT  BODY 
IN  THE  DEVELOPMENT  AND  CONDUCT  OF  A  COEA? 

•IMPACT:  THE  OVERSIGHT  BODY  SHOULD  ASSIST  IN  THE 
DEFINITION 


DECISION  MAKER  ISSUES 


-  IS  A  NEW  START  WARRANTED'5 
•  WHAT  DRIVES  SYSTEM  EFF  ? 

-  WHAT  DRIVES  COST? 

■  WHAT  DRIVES  THE  MS 
DECISION  SCH.? 


STUDY  PLAN 
EXAMPLES 

-  ALTERNATIVES 

-  SCENARIOS/MODELS 

-  THREAT  CAPABILITY 

-  MOEs 

-  ASSUMPTIONS 

-  TASKING  AUTHORITY 

-  APPROACH/METHODOLOGY 


•  IN  EACH  CASE,  WHAT  TRADE-OFFS  SHOULD  BE  INVESTIGATED? 


•POLICY:  None  -RESEARCH  AREA:  N/A 


•ACTION:  N/A 
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•  ISSUE:  How  to  coordinate  COEA  contributors? 


COEA  MANAGEMENT  | 


ACTIVE  PARTICIPATION  OF  BOTH  USER  AND 
ACQUISITION  COMMUNITIES  IS  ESSENTIAL 
TO  SUCCESSFUL  COMPLETION  OF  A  COEA! 


COEA  MANAGEMENT  | 


•  ISSUE:  How  to  make  COEA  responsive? 

•  IMPACT: 

-May  make  follow-on  COEA  as  extensive  as 
predecessor  (unnecessarily  spend  resources, 
$$,  people,  time) 

-May  have  to  restart/cancel  program 
•POLICY:  N/A 

•  RESEARCH  AREA:  N/A 

•  ACTIONS:  Structure  original  COEA  broadly 

enough  via  parametric  bounding-- 
look  at: 

-ranges  of  threat  and  scenarios,  levels  of 
effectiveness 
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MORS  COEA 
MINI-SYMPOSIUM 
MARCH  9-11,  1992 


FUTURE  DIRECTIONS 
WORKING  GROUP 


CHAIRMAN 
MIKE  BAUMAN 
US  ARMY  TRAC 


ISSUE: 


SHOULD  COEA  BE  AVAILABLE  TO  SOURCE  SELECT  AUTHORITY 


POINTS: 


-  SS  CRITERIA  CONSIDER  PGM  FACTORS  BEYOND  COST  EFF 

-  IF  COEA  IS  SS  CRITERIA,  MUST  BE  RELEASED  TO  BIDDERS 

(SUBJECT  TO  LEGAL  INTERPRETATION) 

-  COEA  MAY  CONTAIN  PROPRIETARY/COMPETITION  SENSITIVE 

INFO.  WHEN  SANITIZED,  LIMITS  COEA  UTILITY 


IMPACT  ON  COEA: 


-  MAY  COMPLICATE  EXECUTION  (BIDDERS  REVIEW  MSS) 

-  POTENTIALLY  EXPANDS  SCOPE  BEYOND  COEA  ISSUES 


POLICY  IMPLICATIONS: 


RESEARCH  TOPICS  :  N/A  FOR  OPERATIONS  RESEARCH 


REQUIRED  ACTIONS: 


-  LEGAL  RULING  AS  to  WHETHER  COEA  (W/PROPRIETARY  INFO) 

.  CAN  BE  PROVIDED  TO  SSA  W/O  MANDATORY  RELEASE  TO  3IDDERS 
OR  W/O  SANITIZING  (LEAD:  SERVICE  DEPT  HQ) 
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1  GROWING  UNCERTAINTIES  -  BUT  ANALYSIS  NOT  EVIDENT 

REQ’D  ORIGINALITY  NOT  EVIDENT  -  TOOLS  INADEQUATE 

FLEXIBILITY/TRAINING  MUST  BE  RECONCILED  WITH 
NEED  FOR  ACCEPTANCE/CREDIBILITY 

MORE  REQUIREMENTS  -  BUT  FEWER  RESOURCES  ! 

SHOULD  DO  MORE  JOINT  COEA  -  POOL  RESOURCES  ? 

ALL  PARTIES  WITH  VESTED  INTEREST  MUST  GET  CHANCE 
TO  PARTICIPATE 

WHAT'S  IMPACT  OF  “NON-PROCUREMENT'  ON  COEA  ? 
WHAT'S  PROPER  ROLE  FOR  CONTRACTOR  IN  COEAs  ? 

SCOPE  LIMITED  -  E.G.,  NO  “WHAT  1FS"  ABOUT  DEPLOYMENT  ? 


VARIETY  OF  CONCERNS  (CONT) 


THREAT  ?  THREAT  ??  THREAT  ??? 

-  LONG  RANGE  ESTIMATE 

-  HOW  DO  WE  BUILD  REQ'D  DETAIL  IN  BROAD  ESTIMATE? 

-  HOW  DO  WE  GAIN  CONSENSUS  ON  THE  COEA  THREAT? 

HOW  DO  WE  APPLY  “CAT  l/CAT  II  PROCEDURES"  TO  lll/IV? 

COEA  CONTEXT  FOR  NON-THREAT  DRIVEN  SYSTEMS 

SHOULD  COEA  ADDRESS  AFFORDABILITY?  IF  YES,  HOW? 

HOW  DO  WE  CONVINCE  DECISION  MAKERS  THAT 
LEVEL  OF  DETAIL  MUST  DECREASE? 

WHAT  ARE  PITFALLS  OF  COEA  PROCESS  AND 
HOW  AVOIDED? 
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CHAPTER  5 
REPORT  OF  THE 
SYNTHESIS  WORKING  GROUP 

Clayton  J.  Thomas 


This  report  of  the  Synthesis  Group  (S  YNGRP) 
to  the  COEA  Mini-Symposium  at  Newport  represents 
the  combined  views  of  all  twelve  group  members.  Each 
of  us  is  a  military  operations  research  analyst  and  we 
circulated  around  the  sessions  looking  for  military  OR 
issues.  This  is  a  summary  of  our  findings: 

I  introduced  the  synthesis  group  concept  and 
our  membership.  Jeny  Goldschmidt,  Navy  Sponsor's 
representative  to  MORS,  summarized  effectiveness 
issues  including  those  bearing  on  the  linkage  of  test  and 
evaluation  (T&E)  to  COEAs.  Gerry  McNicols,  CEO  of 
Management  Consulting  and  Research,  summarized 
cost  analysis  issues.  Finally,  Gene  Visco,  Army 
Sponsor's  representative  to  MORS,  presented 
integration  issues  that  are  tied  to  both  cost  and 
effectiveness  estimation.  He  also  included  ideas  for 
future  directions  in  the  performance  of  COEAs. 

S YNGRP  Outline 

I'll  explain  the  purpose  of  the  synthesis  group, 
its  membership  and  the  three  issues  we  focused  on: 
effectiveness,  cost  and  integration. 

S YNGRP  Purpose 

The  purpose  of  the  SYNGRP  was  to  focus  on 
MOR  issues.  Administrative  aspects  of  COEAs  were 
not  of  direct  interest.  Of  special  interest  were  examples 
that  pointed  to  a  need  for  new  OR  methodologies  or 
better  use  of  existing  techniques. 

SYNGRP  Members 

The  members  of  the  SYNGRP  are  listed  here. 
Among  these  members  there  is  a  depth  of  MOR 
experience  in  cost  and  effectiveness  estimation  and  in 
service  and  agency  perspectives.  Half  of  the  SYNGRP 
had  served  in  a  similar  capacity  at  a  previous  mini¬ 
symposium. 

SYNGRP  Posts 

We  positioned  our  members  to  sample  as 
many  discussions  and  perspectives  as  possible.  We  had 
at  least  one  member  in  each  of  the  working  groups  and 


in  three  working  groups  there  were  two.  Additionally, 
three  SYNGRP  members  were  in  rotating  posts. 
Thomas  followed  the  Topic  1,  6,  and  7  experts.  Visco 
shadowed  Topic  2,  5,  and  9  experts.  Goldschmidt 
followed  Topic  3, 4,  and  8  experts. 

Effectiveness  Issues 

Joint  COEAs  reflect  joint  operations  and/or 
consider  options  from  more  than  one  service.  We  have 
experienced  many  difficulties  in  evaluating  options  that 
involve  more  than  one  component  of  a  single  service. 
These  are  compounded  and  intensified  in  joint  COEAs 
limitations  of  current  service  models,  data  bases,  and 
experience,  suggest  that  joint  COEAs  may  require 
contributions  from  joint  staff  and/or  more  than  one 
service. 

New  technology,  such  as  "DIS"  (Distributed 
Interactive  Simulation)  permits  greater  operational 
realism  and  flexibility  in  military  simulations.  To  derive 
full  benefits  from  such  technological  advances  and  to 
permit  sharing  of  separate  data  bases,  we  must  establish 
suitable  standards  and  protocols,  and  DMSO  —  The 
Defense  Modeling  and  Simulations  Office  —  can  help 
in  this. 

Guidance  for  COEAs  emphasizes  several 
''linkages. "  First  is  linkage  of  measures  of  engineering 
performance  (MOPS)  to  intermediate  measures  of 
"encounter”  effectiveness  (MOES)  to  high  level 
campaign  measures  of  operational  outcomes  (MOOS). 
We  can  test  MOPS  and  sometimes  MOES  but  very 
often  we  can  evaluate  MOES  and  almost  always 
MOOS,  only  through  some  kind  of  simulation.  Second 
is  linkage  of  effectiveness  measure  estimates  and  cost 
estimates  to  ensure  that  they  apply  to  the  same  system. 

Much  recent  guidance  for  COEAs  deals  with 
requirements  for  campaign  analysis.  Even  defining 
these  poses  difficulties  and  accomplishing  them  leads 
to  many  more.  If  they  could  be  well  performed  and 
adequately  reflect  a  commander's  operational  art,  they 
would  greatly  enhance  the  values  of  COEAs.  Some 
current  efforts  hold  promise,  but  many  further  steps  are 
needed  to  achieve  potential. 
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Cost  Issues 


Cross  training  of  effectiveness  analysts  and 
cost  analysts  could  help  those  in  each  community,  but 
especially  effectiveness  analysts.  Now,  e.g.,  many 
effectiveness  analysts  see  the  need  to  carefully  define 
the  parameters  of  their  estimates,  but  may  ask  "what  is 
the  cost"  without  regard  to  quantity',  constant  or  then- 
year  dollars,  force  or  system  content,  etc.  Thus  cost  is 
often  the  forgotten  half  of  cost-effectiveness  analysis. 

Better  macro  level  parametric  cost  models 
would  permit  quick  reaction  cost  tradeoffs  in  early 
stages  of  analysis  and  the  quick  evaluation  of  enough 
options  to  better  ensure  the  identification  of  better 
alternatives. 

Better  risk  and  uncertainty  analysis  models 
would  lead  to  better  identification  of  cost  drivers  to 
robust  COEAs  and  thus  to  better  decision  support. 

A  new  acquisition  strategy,  putting  systems 
"on  the  shelf'  after  development,  will  require  cost 
estimating  methodologies  to  put  more  weight  on  R&D 
costs  vs  life  cycle  costs. 

Different  speakers  in  the  mini-symposium 
disagreed  on  how  to  couple  cost  and  effectiveness 
estimates  in  COEAs.  Research  is  needed  to  develop 
definition  or  new  approaches  as  a  basis  for  clarifying 
policy. 

For  a  cost  estimate,  a  cost  analyst  currently 
writes  a  "card"  (cost  analysis  requirement  document)  to 
define/specify  the  system  or  force  under  study.  Should 
a  COEA  always  require  such  a  CARD  and  should 
effectiveness  analysts  also  play  a  role? 

Integration  Issues 

It  is  trite  to  note  the  culture  shock  of  dramatic 
threat  changes,  burgeoning  technology,  and  shrinking 
budgets,  but  the  shock  is  nonetheless  real.  The  analytic 
community  has  tremendous  opportunities  but  faces 
unprecedented  challenges;  integrating  these  is  issue 
one. 

Meeting  the  challenges  requires  true  thought 
and  defies  "cookbook  recipe."  In  particular,  we  must 
see  models  as  tools  and  not  as  panaceas.  We  must  resist 
those  who  misuse  "models"  as  an  icon,  to  the  neglect  of 
a  concern  for  data  and  well  structured  analysis. 


The  challenges  include  not  only  the  linkage  of 
measures  and  of  cost  to  effectiveness  but  also  higher 
level  questions  of  affordability  and  sustainability.  The 
question  of  affordability  transcends  a  COEA  for  a 
single  system  and  involves  consideration  of  multiple 
systems  competing  for  scarce  dollars  but  may  be 
painfully  relevant  to  system  acquisition.  Sustainability 
may  refer  to  troops  on  a  battlefield  or  a  nation's 
industrial  base  but  each  is  essential  to  support  of 
national  strategies  and  therefore  an  essential 
consideration  in  system  and  force  decisions. 

More  use  of  joint  and  combined  forces,  the 
changing  density  of  engaged  troops  and  the  new 
technologies  all  require  new  techniques  and 
development  in  decision  art  and  science. 

To  some  of  us,  this  implies  more  use  of 
traditional  interdisciplinary  teams  and  increased 
interrelationships  among  analysis  groups  and 
institutions. 

EFFECTIVENESS  ISSUES 
Potential  OR  Problems 

As  Clayton  Thomas  just  noted,  our  group 
focused  on  effectiveness  and  T&E  issues.  We  also 
followed  modeling  and  tools  (Sam  Gardiner), 
operational  effectiveness  methodologies  (Tony 
Brinkley),  and  linkage  of  performance  effectiveness 
(Lese,  Ledesma  and  Seglie).  We  focused  on 
effectiveness  methodology  and  linkage.  We  developed 
four  issues  based  on  a  combination  of  what  participants 
in  this  mini-symposium  articulated  and  our  own 
personal  views. 

EFfectiveness  Issue  1 

Our  first  issue  deals  with  jointness  in 
effectiveness  analysis.  The  future  directions  working 
group  made  excellent  comments  on  this  issue  and  raised 
the  problem  of  how  to  oversee  and  provide  oversight. 
We  have  similar  concerns. 

By  "joint  COEAs"  we  are  referring  to  COEAs 
that  reflect  operations  and  that  consider  options  from 
more  than  one  service.  We  have  seen  problems  within 
single  services  with  respect  to  options  that  consider 
more  than  one  component  of  the  service  and  recognize 
that  this  is  a  difficult  thing  to  do.  We  believe  some 
thought  should  be  given  to  how  well  a  single  service 
can  do  a  joint  COEA  and  what  role,  if  any,  JCS  and  the 
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CINCs  might  have  with  respect  to  joint  COEAs.  We 
also  believe  current  service  models  may  not  be 
adequate  to  consider  jointness  because  most  of  them 
focus  on  single  service  operations.  And,  jointness  is 
related  to  the  issues  of  campaign  analysis  because 
campaign  models  must  reflect  joint  operations. 

We  see  several  long  range  impacts  related  to 
proper  consideration  of  jointness  in  our  COEA 
effectiveness  analysis.  These  include  potential  for 
cheaper  solutions,  potential  for  more  operational 
capabilities,  better  ties  between  the  acquisition  process 
and  how  we  fight,  and  more  inter-service  analyst 
exchanges  similar  to  the  CAA/AFSA  exchange 
mentioned  by  Walt  Hollis.  As  RADM  Allen  noted 
yesterday,  if  people  don't  get  on  board  (with  respect  to 
jointness)  they  will  be  left  behind. 

Effectiveness  Issue  2 

Our  second  issue  deals  with  the  use  of  new 
technology  to  improve  modeling.  Today,  we  can  do 
things  we  would  not  have  dreamed  of  ten  years  ago.  For 
example,  today  DIS  allows  us  to  conduct  simulations 
that  include  widely  dispersed  participants,  including 
operational  units  from  several  services,  thereby  leading 
to  more  operational  realism  in  these  joint  operations. 
We  also  believe  that  by  establishing  standards  and 
protocols,  it  will  allow  us  to  address  specific  issue 
thereby  allowing  us  to  deal  with  uncertainty  in  a  much 
better  fashion  than  we  can  today.  It  is  also  important  to 
develop  our  systems  so  we  can  share  data  bases. 
Finally,  we  must  work  with  the  DMSO  on  these  issues 
because  they  are  heavily  involved  with  this  issue  and  if 
these  items  are  not  properly  defined,  they  could  be 
counter  productive. 

We  see  the  potential  for  significant  impact 
from  the  proper  use  of  new  technologies  in  the  areas  of 
increased  flexibility,  reduced  costs,  enhanced  efficiency 
and  more  operational  realism. 

Effectiveness  Issue  3 

The  T&E  working  group  made  excellent 
comments  on  the  issue  of  linkage  between  COEAs  and 
T&E.  It  noted  that  we  must  develop  criteria  to  establish 
this  linkage  to  ensure  that  modeling  and  simulation 
techniques  used  for  COEAs  also  support  the  T&E  for 
the  system  as  it  is  developed. 

As  analysts  involved  in  COEAs,  we  must 
interface  with  the  T&E  community  to  develop  an 


understanding  of  the  types  of  tests  they  will  conduct  so 
the  methodologies  developed  and  used  for  COEAs  and 
T&E  are  comparable  and  estimate  the  same  MOEs. 
Further,  the  methodology  must  establish  relationships 
between  the  MOEs/MOPs  used  for  T&E  so  that  it  is 
possible  to  show  how  changes  in  lower  level  measures 
drive  the  higher  level  measures  (e.g.,  aircraft  losses) 
that  often  cannot  be  measured  through  T&E  because  of 
cost  or  loss  of  life.  These  relationships  can  then  be  used 
by  the  T&E  community  to  examine  sensitivities  of 
variables  they  cannot  measure  to  other  variables  to 
determine  which  ones  have  a  major  impact  on 
operational  outcomes.  These  results  can  be  used  by  the 
T&E  community  to  select  appropriate  variables  for 
testing. 

We  see  the  potential  for  significant  impact 
from  the  proper  linkage  between  COEAs  and  T&E 
through  more  credible  and  consistent  COEAs  and 
testing  and  systems  which  serve  warfighting  needs. 

Effectiveness  Issue  4 

The  effectiveness  working  group  made 
excellent  comments  on  the  issue  of  campaign  analysis. 
We  agree  that  it  is  tough  to  define  campaign  analysis. 
Guidance  is  needed. 

First,  campaign  analysis  means  different  things 
to  different  people.  We  must  develop  a  common 
definition.  Second,  we  develop  guidance  regarding 
when  analyses  are  appropriate.  We  clearly  saw  their 
value  in  conducting  mission  area  analysis  (MAA)  to 
determine  requirements,  but  did  not  see  a  clear  value  in 
the  later  milestones  for  system  development.  Other 
questions  that  must  be  answered  include  who  does 
them,  how  do  we  establish  credibility  and  what  are  the 
proper  MOEs. 

We  also  heard  comments  on  this  issue  from 
two  of  our  senior  panel  members.  MG  Ralston  noted 
that  linking  models  to  campaign  output  is  a  weak  area. 
Mr.  Hollis  noted  that  we  need  new  tools  and  methods  to 
allow  us  to  conduct  campaign  models  properly. 

We  see  the  potential  for  significant  impact 
from  proper  use  of  campaign  analysis  through  more 
consistent  COEAs,  better  focus  on  military  outcomes 
and  better  rationale. 
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Cost  Issues 

Let  me  note  that  some  cost  analysts  believe 
they  have  divine  guidance  by  quoting  the  New 
Testament  verse:  LUKE  14:28. 

We  identified  the  following  six  cost  issues: 

1)  cross-training, 

2)  the  need  for  better  macro  level  parametric 
cost  models, 

3)  the  need  for  better  risk  and  uncertainty 
analysis  models, 

4)  the  lack  of  a  methodology  to  properly 
weight  and  cost  R&D  if  systems  are  "put  on 
the  shelf'  after  development  as  the  potential 
new  acquisition  strategy  rnay  require, 

5)  the  need  for  new  techniques  for  cost 
effectiveness  comparison  and  coupling, 

6)  the  need  for  and  use  of  a  cost  analysis 
requirements  document  (and  who  should 
develop  it). 

Cross  Training 

Most  effectiveness  analysts  vr!  not  answer 
the  question  "what  is  the  effectiveness"  without  asking 
for  clarification,  e.g.,  at  what  altitude,  dive  angle, 
intervelometer  setting,  etc.  But,  they  will  then  ask 
"what  is  the  cost"  without  regard  to  quantity,  constant 
or  then-year  dollars,  force  or  system  content,  etc.  Thus 
cost  is  typically  the  forgotten  half  of  cost-effectiveness 
analysis. 

Often  the  cost  analysis  is  not  brought  into  the 
early  stages  of  COEA  planning,  yet  cost  issues  could  be 
anticipated,  and  the  cost  team  could  begin  developing 
relevant  data  bases  if  they  were  "in  at  the  beginning." 

Impact  on  Analysis:  The  key  impact  on  the 
analysis  is  that  the  costing  is  not  of  high  quality  when 
hurried.  In  addition,  design  trades  by  the  cost  team  may 
lead  to  useful  and  better  alternatives  which  can  be 
added  to  the  COEA. 

Impact  on  Policy:  None 


Research  Areas:  The  development  of  a  cost 
methodology  training  course  useful  to  all  COEA  participants. 

Actions:  An  executive  overview  training 
course  should  be  developed  which  would  be  used  to 
familiarize  all  COEA  team  participants  with  cost, 
technical,  and  test  issues  relevant  to  the  COEA.  Initial 
team  training  should  be  an  integral  part  of  the  COEA 
startup.  A  joint  education  conference  between  MORS 
and  the  Society  of  Cost  Estimating  and  Analysis 
(SCEA)  should  be  held. 

Macro  Level  Parametric  Cost  Models 

Cost  analysis  is  often  required  very  late  or 
very  early  in  the  COEA  process.  There  is  typically 
insufficient  time  to  apply  detailed  costing  methodology. 

In  addition,  parametric  cost  models  which  are 
performance  parameter  based  are  needed  to  allow 
various  cost  tradeoff  analyses  to  be  performed. 

In  the  effectiveness  side,  measures  of 
performance  (MOP)  have  to  relate  to  measures  of 
operational  output  (MOO).  It  would  be  useful  to  have 
measures  of  cost  (MOC)  to  relate  to  both  of  these.  Such 
measures  might  include  flyaway  cost,  weapon  system, 
cost,  acquisition  cost,  program  cost,  unit  cost,  cost  per 
kill,  etc. 

Impact  on  Analysis:  More  macro  level  models 
would  allow  quick  reaction  cost  tradeoffs  to  be 
conducted  in  the  early  stages  of  analysis  to  reduce  the 
number  of  alternatives  or  to  permit  more  "analysis"  of 
costs  rather  than  being  spent  on  cost  estimate 
generation. 

Impact  on  policy:  None 

Research  Areas:  Development  of  macro  level 
parametric  cost  models. 

Actions:  Develop  new  "quick  response" 
methodologies  or  models  with  performance  parameters. 
These  would  have  to  be  sensitive  to  acquisition  phase 
and  be  unique  for  each  commodity  class. 

Risk  And  Uncertainty  Models 

A  critical  methodology  problem  is  to  identify 
the  key  "cost  drivers"  for  each  alternative.  Such  cost 
drivers  should  relate  to  the  system  performance  or 
physical  parameters.  Each  cost  driver,  however,  is 
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subject  to  risk  and  uncertainty.  Probability  distributions 
can  be  developed  and  these  can  be  aggregated  to  obtain 
a  total  cost  uncertainty.  New  models  to  handle  such 
uncertainty  are  needed.  The  analytical  methodology 
exists. 

Impact  on  Analysis:  COEA  would  be  more 
robust  if  uncertainties  were  explicitly  handled. 

Impact  on  Policy:  Determining  how  such 
uncertainties  in  both  cost  and  effectiveness  should  be 
treated. 

Research  Areas:  The  key  research  needed  is  to 
review  the  appropriate  methodologies  in  the  literature 
and  develop  new  models  to  incorporate  it. 

Actions:  Development  of  quick  response 
methodologies  and  new  risk  models  useful  to  the 
acquisition  phase  and  commodity  class.  Standard 
presentation  formats  are  needed  to  illustrate  the 
importance  of  uncertainty  in  evaluating  the  selection  of 
alternatives. 

Costing  And  Weighing  R&D 

New  techniques  to  estimate  R&D  are  needed. 
The  current  "cost  drivers"  of  research  and  prototype 
development  are  time  and  level  of  engineering  effort 
(eg.,  size  of  engineering  team).  Current  R&D  cost  data 
bases  are  poor,  particularly  with  many  recent  firm 
fixed  priced  contracts  allowing  little  or  no  cost 
visibility.  It  is  also  expensive  to  startup  or  reopen  a  cold 
production  line  since  prototype  systems  may  have  been 
built  with  "soft"  vs  "hard"  tooling.  This  could  even 
result  in  the  disappearance  or  production  lines 
altogether. 

Impact  on  Analysis:  The  analysis  would  be 
based  on  a  cost  different  from  life  cycle  cost. 

Impact  on  Policy:  The  policy  change  would  be 
to  revise  the  LCD  requirement  in  5000.2 

Research  Areas:  New,  robust  methods  for 
determining  R&D  costs  are  needed. 

Actions:  The  required  actions  are  to  develop 
new  data  bases  and  then  new  methodologies  and 
models  from  this  data. 


Coupling  Cost  And  Effectiveness 

There  was  some  confusion  during  the 
conference  as  to  the  proper  comparisons.  For  example, 
one  speaker  said  do  not  combine  cost  and  effectiveness, 
but  treat  them  separately.  Another  said  use  the 
analytical  hierarchy  process  or  utility  theory.  Dr.  Chu 
said  not  to  be  artificially  constrained  to  equal  cost  or 
equal  effectiveness  alternatives.  The  Army  commonly 
uses  force  cost  comparisons  while  the  Air  Force  and 
Navy  use  systems  and/or  comparisons.  Speakers  talked 
about  using  "decision  cost"  versus  life  cycle  cost 
comparisons  and  noted  the  consistency  issue  with 
baseline  cost  estimates  (BCE)  or  independent  cost 
estimates  (ICE).  Others  noted  that  cost  is  peace-time 
driven  while  effectiveness  is  wartime  driven. 

Impact  on  Analysis:  Define  "upfront"  what  the 
assumptions  and  ground  rules  are  for  COEA. 

Impact  on  Policy:  A  policy  issue  is  to  clarity 
what  "decision  costs"  are  relevant. 

Research  Areas  and  Actions:  Research  needs 
to  be  conducted  to  develop  the  appropriate  definitions, 
clarifications  or  new  approaches. 

Developing  Cards 

For  a  cost  estimate,  a  cost  analyst  currently 
writes  CARD,  but  should  the  effectiveness  analyst 
write  CARD  for  the  COEA.  Should  a  COEA  have  a 
CARD  which  is  currently  only  required  for  an 
ICE/BCE/POE? 

Impact  on  Analysis:  Specification  and 
definition  of  alternative  systems/forces  is  critical  to  the 
costing  exercise. 

Impact  on  Policy:  A  guidance  document  for  a 
COEA  would  be  a  policy  addition. 

Research  Areas/Action:  Write  a  directive 
specifying  the  structure  and  contents  of  a  CARD 
equivalent  document. 

INTEGRATION 

Culture  Shock 

During  the  last  two  days,  the  synthesis  group 
has  observed  the  collective  sense  of  culture  shock 
invading  the  analysis  community.  Our  traditional,  well- 
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understood  and  well-analyzed  threat  has  been 
uncooperative.  Great  technological  opportunities  loom 
for  the  future  (eg  SDI,  non-lethal  weapons),  concurrent 
with  these  new  forces,  resources  (money  and  people) 
are  seriously  reduced. 

Related  to  these  new  forces,  the  Department  of 
Defense  is  adopting  new  management  techniques  (e.g. 
TQM)  and  processes  (e.g.,  systems  acquisition,  DMR, 
CMI)  to  increase  efficiency  in  providing  for  the 
national  defense. 

The  analytic  community  faces  unprecedented 
demand  and  challenges  to  its  capabilities. 

Models  Not  Synonymous  With  Analysis 

There  has  been  much  talk  about  the  tools  and 
materials  of  the  analyst  —  the  models  and  the  data 
bases  that  feed  them.  One  position  holds  that  we  need 
more  campaign  analyses  while  another  argues  against 
the  time  and  expense  involved  in  analyzing  increasing 
numbers  of  scenarios.  The  latter  group  demands  more 
quick-tumaround  tools  and  top-down  thinking. 

Above  all,  it  is  important  to  remember  that  the 
emphasis  here  is  on  analysis  —  thinking  to  help  the 
decision  maker.  However,  much  we  may  desire  it,  there 
will  not  be  a  cookbook  solution  to  the  varied  and 
multifaceted  problems  we  will  confront.  The 
ingredients  and  the  utensils  for  each  will  be  different; 
our  job  is  to  make  the  finished  product  palatable  or  at 
least  edible. 

We  therefore  strongly  advise  our  colleagues  to 
focus  on  that  heavy  investment  (40%  of  the  total  study 
time)  upfront  for  study  planning,  formulating  the 
problem,  obtaining  the  data,  gathering  the  tools...  and 
reacquainting  ourselves  with  existing  applicable 
techniques  such  as  experimental  design,  sensitivity 
analyses  and  decision  science  to  reduce  uncertainty  and 
deal  with  the  risk  increasing  importance  to  our  analysis. 

Analytic  Challenges 

The  new  challenges  are  illustrated  most 
emphatically  by  what  has  been  termed  here  as  linkage, 
the  most  obvious  and  familiar  linkage  is  cost  with 
effectiveness.  Another  is  effectiveness  to  system 
measures  that  can  be  tested  and  translated  to 
operational  outcomes  on  the  battlefield.  A  third  (that 
many  would  prefer  to  ignore)  is  that  of  system 
effectiveness  to  allocation  decisions  —  a  vitally  needed 


linkage.  And,  we  have  the  problem  of  linkage  in  time 
—  the  problem  of  reconstitution,  both  of  our  industrial 
base  in  the  event  of  major  war  in  the  future  and  of 
forces  on  the  battlefield  in  the  sense  of  tactical  or 
operational  levels. 

The  community  must  focus  on  the 
decisionmakers  and  the  way  to  effectively  provide  them 
with  information  ,  which  may  be  a  lesson  in  decision 
science.  There  are  "tactical"  challenges  also,  for 
example,  developing  better  understanding  of  the 
nonlinearaities  of  the  battlefield  or  the  changed 
densities  (a  challenge  that  seems  to  be  a  continuous 
one). 

All  the  forgoing  implies,  to  some  of  us  at  any 
rate,  a  return  to  the  past;  greater  use  of  the  true, 
traditional  interdisciplinary  teams;  increased 
interrelationships  among  analysis  group  and 
institutions;  further  centralization  of  analysts. 

Future  Responsibilities 

Despite  present  capacities  to  overcome 
difficult  obstacles  (to  the  present  time,  at  least),  we 
acknowledge  the  need  for  improvement.  The  multi¬ 
disciplinary  communities  must  work  together.  In  some 
ways  there  are  many  opportunities  unfolding;  the  OSD 
initiative  on  modeling  and  simulations  has  a  potential 
for  improving  linkage,  at  least  in  communications 
among  and  between  the  services,  the  Joint  Staff  and 
OSD  and  possibly  influencing  standards. 

As  pointed  out,  models  and  simulations  are  not 
substitutes  for  analysis  —  They  are  only  particular 
tools.  The  military  operations  research  community  must 
improve  its  credibility  with  decision  makers.  One  small 
step  is  to  increase  the  transparency  of  our  analysis, 
particularly  the  analysis  that  relies  heavily  or 
exclusively  on  computer  models.  Another  contribution 
to  credibility  comes  from  improvements  in  verification, 
validation  and  accreditation  process  through  events  like 
the  MORS  Simval  series  of  special  meetings. 

Improvements  can  not  come  solely  from  the 
efforts  of  the  Services  and  the  analysts  regulations  and 
guidelines  must  be  carefully  thought  through.  One 
recommendation  the  synthesis  group  makes  is  that  an 
independent  study  be  commissioned  to  define  how  the 
systems  acquisition  and  resource  allocation  processes 
should  work  in  this  unprecedented  era;  perhaps  an 
FFRDC  could  be  charged  with  the  responsibility. 
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SYNTHESIS  GROUP 
OBSERVATION  POSTS 

POSTS 

OBSERVERS 

WG  1 ,  TEST  AND  EVALUATION 

GUSTAFSON 

WG  2,  THREAT  ASSESSMENTS 

SCHNURRPUSCH 

WG  3,  EFFECTIVENESS  METHODS 

BEXFIELD  &  KAYS 

WG  4,  COSTING  METHODS 

McNICOLS  &  RHODE 

WG  5,  COEA  MANAGEMENT 

BETTENCOURT 

WG  6,  FUTURE  DIRECTIONS 

HENNINGSEN  &  SMITH 

TOPICS:1  (LESE),  2  (CASSADY), 

3  (FEIGLEY) 

THOMAS 

TOPICS  2  (FAUSS),  S  (NUSSBAUM), 

9  (DIAZ) 

VISCO 

TOPICS:3  (GARDNER),  4  (GARDNER), 

8  (LESE,  LEDESMA,  SEGLIE) 

GOLDSCHMIDT 

EFFECTIVENESS  ISSUES 


JOINT  COEAs 

NEW  TECHNOLOGIES: 

DISTRIBUTED  INTERACTIVE  SIMULATION 

DATA  SHARING 

STANDARDS 

DMSO 

LINKAGE: 

MOPs/MOEs/MOOs,  EFFECTIVENESS 
AND  COST 

COMPAIGN  ANALYSIS 
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ISSUE:  PRACTICE 
CONTINUED 


RESEARCH  AREAS:  COST  METHODOLOGY 
TRAINING  COURSE  FOR  COEA  PARTICIPANTS 


ACTIONS: 

•  DEVELOP  EXECUTIVE  OVERVIEW  TRAINING 
COURSE  COVERING  ALL  ELEMENTS  OF  ANALYSIS 

•  CONDUCT  INITIAL  “TEAM”  TRAINING  AT  THE 
START  OF  A  COEA 

•  HOLD  JOINT  EDUCATION  CONFERENCE 
BETWEEN  MORS  AND  SOCIETY  OF  COST 
ESTIMATING  AND  ANALYSIS  (SCEA) 


METHODOLOGY 

ISSUE:  BETTER  MACRO  LEVEL  PARAMETRIC 
COST  MODELS  NEEDED 

DISCUSSION: 

•  COST  ANALYSIS  IS  OFTEN  REQUIRED  VERY 
EARLY  AND  VERY  LATE  IN  THE  COEA  PROCESS 
WHERE  INSUFFICIENT  TIME  IS  AVAILABLE  FOR 
DETAILED  COSTING  METHODOLOGY 

•  PARAMETRIC  MACRO  COST  MODELS  - 
PERFORMANCE  PARAMETER  BASED  -  ARE 
NEEDED  TO  ALLOW  EARLY  COST  TRADES 

•  MEASURES  OF  PERFORMANCE  (MOP)  ARE  TO 
RELATE  TO  MEASURES  OF  EFFECTIVENESS  (MOE) 
AND  MEASURES  OF  COST  (MOC)  OUGHT  TO 
RELATE  TO  BOTH 

IMPACT  ON  ANALYSIS:  QUICK  REACTION 
TRADEOFFS  WOULD  PERMIT  MORE  "ANALYSIS” 

OF  COST  VERSUS  GENERATION 

IMPACT  ON  POLICY:  NONE 
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METHODOLOGY 

CONTINUED 


RESEARCH  AREAS:  DEVELOP  PARAMETRIC 
MACRO  MODELS 

ACTIONS: 

•  DEVELOP  QUICK  RESPONSE 
METHODOLOGIES,  NEW  MACRO  LEVEL 
MODELS  WITH  PERFORMANCE 
PARAMETERS  AVAILABLE  BY  ACQUISITION 
PHASE,  AND  BY  COMMODITY  CLASS 

•  HOLD  JOINT  EDUCATION  CONFERENCE 
BETWEEN  MORS  AND  SOCIETY  OF  COST 
ESTIMATING  AND  ANALYSIS  (SCEA) 


METHODOLOGY 


ISSUE:  BETTER  RISK  AND  UNCERTAINTY 
ANALYSIS  NEEDED 

DISCUSSION: 

•  COST  ANALYSIS  TEAM  NEEDS  TO  IDENTIFY 
KEY  “COST  DRIVERS"  WHICH  RELATE  TO  SYSTEM 
PERFORMANCE  OR  PHYSICAL  PARAMETERS 

•  THESE  “COST  DRIVERS”  ARE  SUBJECT  TO  RISK 
OR  UNCERTAINTY  (THIS  IS  DIFFERENT  FROM 
SENSITIVITY  ANALYSIS) 

•  APPROPRIATE  DISTRIBUTIONS  AND 
AGGREGATION  MODELS  ARE  NEEDED 

IMPACT  ON  ANALYSIS:  BETTER 
UNDERSTANDING  IN  COSTS 

IMPACT  ON  POLICY:  HOW  SHOULD 
UNCERTAINTY  IN  COST  AND  EFFECTIVENESS 
CALCULATIONS  BE  TREATED? 
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METHODOLOGY 
RISK  (CONT) 


RESEARCH  AREAS: 

•  DEVELOP  APPROPRIATE  METHODOLOGY 
AND  MODELS 


ACTIONS: 

•  DEVELOP  QUICK  RESPONSE  METHODOLOGIES, 
NEW  RISK  MODELS  BY  ACQUISITION  PHASE,  AND 
BY  COMMODITY  CLASS 

•  DEVELOP  STANDARD  PRESENTATION  FORMATS 
TO  ILLUSTRATE  IMPORTANCE  OF  UNCERTAINTY 
TO  ALTERNATIVE  SELECTION 


METHODOLOGY 

ISSUE:  METHODOLOGY  TO  COST  AND  PROPERLY 
WEIGHT  R&D  COST  (vs  LCC)  IF  SYSTEMS  PUT  “ON 
SHELF”  AFTER  DEVELOPMENT  (POTENTIAL  NEW 
ACQUISITION  STRATEGY) 

DISCUSSION: 

•  NEW  TECHNIQUES  TO  ESTIMATE  R&D  COST 
NEEDED 

•  CURRENT  “COST  DRIVERS”  OF  R&D  IS  TIME 
(SYSTEMS  ENGINEERING  EFFORT),  SIZE  OF 
RESEARCH  TEAM,  ETC... 

R&D  COST  DATA  BASES  POOR,  FFP  CONTRACT 
HAVE  LITTLE  OR  NO  COST  VISIBILITY 

•  STARTUP  OR  REOPENING  COLD  PRODUCTION 
LINE 

•  COSTLY  (HARD  TOOLING  vs  SOFT  TOOLING)  - 
COULD  RESULT  IN  DISAPPEARANCE  OF 
PRODUCTION  LINES 

IMPACT  ON  ANALYSIS:  DIFFERENT  COST  THAN 
LIFE  CYCLE  COST 

IMPACT  ON  POLICY:  REVISE  LCC  REQUIREMENT 
IN  5000.2 
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METHODOLOGY 
R&D  COST  (CONT) 


RESEARCH  AREAS: 


ROBUST  METHODS  FOR 
DETERMINING  R&D  COSTS 


ACTIONS: 


-  DEVELOP  NEW  DATA  BASES 

-  DEVELOP  NEW  METHODOLOGY 


METHODOLOGY  (CE  COMPARISON) 

ISSUE:  DEVELOP  NEW  TECHNIQUES  FOR  COST 
EFFECTIVENESS  COMPARISON  OR  COUPLING 

DISCUSSION: 

•  ONE  SPEAKER  SAID:  “DO  NOT  COMBINE  COST 
EFFECTIVENESS” 

•  ANOTHER  SAID:  “USE  ANALYTICAL  HIERARCHY 
PROCESS,  UTILITY  THEORY,  ETC,. .TO  COMBINE" 

•  DR  CHU  SAID  “DON'T  ARTIFICIALLY  FORCE 
EQUAL  COST  OR  EQUAL  EFFECTIVENESS 
COMPARISON" 

•  ARMY  USES  FORCE  COST  COMPARISONS  WHILE 
AIR  FORCE/NAVY  USES  SYSTEM/PLATFORM 
COMPARISONS 

•  DECISION  COST  vs  LCC  (ICE  OR  BCE)  SCOPE  OF 
COST  MAY  BE  DIFFERENT  THAN  COEA-CONSISTEN1 

IMPACT  ON  ANALYSIS:  NEED  TO  DEFINE  UP 
FRONT  WHAT  COST  GROUND  RULES  AND 
ASSUMPTIONS  ARE 

IMPACT  ON  POLICY:  NEED  TO  CLARIFY  WHAT 
“DECISION  COSTS"  ARE  RELEVANT 
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METHODOLOGY 
CE  COMPARISON  (CONT) 


RESEARCH  AREAS  AND  ACTIONS: 

DEVELOP  APPROPRIATE  DEFINITIONS, 
CLARIFICATIONS,  NEW  APPROACHES  E.G. 
MATHEMATICAL  PROGRAMMING 


PRACTICE 

CARD 

ISSUE:  CAN  A  COST  ANALYSIS  REQUIREMENTS 
DOCUMENT  (CARD)  BE  DEVELOPED  FOR  COEA? 
WHO  SHOULD  DEVELOP? 


DISCUSSION: 

•  COST  ANALYST  CURRENTLY  WRITES  CARD, 
BUT  SHOULD  EFFECTIVENESS  WRITE  COEA 
EQUIVALENT? 

•  CARD  IS  FOR  ICE/POE/BCE  NOT  COEA 


IMPACT  ON  POLICY:  GUIDANCE  DOCUMENTS 
FOR  COEA  COULD  BE  POLICY  ADDITION 

RESEARCH  AREAS/ACTIONS:  WRITE 
DIRECTIVE  SPECIFYING  STRUCTURE  AND 
OF  CARD  EQUIVALENT  DOCUMENT 
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INTEGRATION 

CULTURE  SHOCK 


DISCUSSION 


•  CHANGING  THREAT 

•  NEW  WEAPON  &  TECHNOLOGY  OPPORTUNITIES 

•  DECLINING  RESOURCES 

•  NEW  MANAGEMENT  TECHNIQUES 

•  NEW  ACQUISITION  PROCESSES 


IMPACT 


•  INCREASING  “DEMAND”  FOR  ANALYSIS 

•  DECLINING  ANALYTIC  RESOURCES 


INTEGRATION 


ANALYTIC  THINKING 

•  POINT  ESTIMATES  vs  RANGES: 

SENSITIVITY  ANALYSIS 

•  OVERARCHING  SCENARIO  vs  MULTIPLE  OPTIONS: 
EXPERIMENTAL  DESIGN  APPLIED  TO  MODELING 

•  QUICK  LOOK  vs  CAMPAIGN  ANALYSIS: 
INTEGRATED  HIERARCHY  OF  MODELS 

•  COOKBOOK  FOR  COEAs  vs  ONE-OF-A-KIND 
COEAs:  PROCEDURAL  CONSISTENCY  &  ANALYTIC 
METHODOLOGY  FLEXIBILITY 

IMPACT 

•  REQUIREMENT  FOR  GREATER  EMPHASIS  ON 
CREATIVE  THINKING  IN  ANALYSIS 

•  DE-EMPHASIS  ON  “COOKBOOK  SOLUTION" 
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INTEGRATION 


DISCUSSIONS 

•  REQUIRED  LINKAGES:  MEASURES,  C&E, 
AFFORDABILITY,  FORCES  (PRESENT&FUTURE), 
RECONSTITUTION  (BATTLEFIELD/INDUSTRIAL  BASE) 

•  CONSIDERATION  FORM  MILITARY  SCIENCE: 
HISTORY,  NON-LINEAR  BATTLEFIELD,  JOINT/ 
COMBINED  INTERAGENCY  OPERATIONS,  SPECTRUM 
OF  CONFLICT 

•  USING  DECISION  SCIENCE:  STRUCTURING 
DECISION  OPTIONS,  INTEGRATION  METHODS, 
PRESENTATION 

IMPACT 

•  DOD  MOVING  TO  EMPHASIS  COMMONALITY: 
REQUIREMENTS,  INTERRELATIONSHIPS  OF 
METHODS,  CENTRALIZATION  OF  RESOURCES 

•  MOR  COMMUNITY  FORCED  TO  RETURN  TO  ITS 
INTER-DISCIPLINARY  FOUNDATION 


INTEGRATION 


IMPROVE 

•  COMMUNICATIONS:  BETWEEN/WITHIN 
COMMUNITIES 

•  PRACTICE:  TRANSPARENCY,  W&A 

•  IMPLEMENTATION:  FINE  TUNE  REGULATIONS  AND 
UNDERSTAND  THE  PROCESS 

IMPACT 

•  SUPPORTS  NEED  FOR  FUTURE  MORS  SPECIAL 
MEETINGS 

NEED 

•  IN-DEPTH  STUDY  OF  ACQUISITION  &  RESOURCE 
ALLOCATION  PROCESS 

•  EXTENSIVE  OR  INVOLVEMENT  IN  DISTRIBUTED 
INTERACTIVE  SIMULATIONS  INITIATIVES 
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POTENTIAL  OR  PROBLEMS 
IN  SUPPORT  OF  COEAs 


TOPICS  INCLUDE: 

•  EFFECTIVENESS  METHODOLOGIES 

•  LINKAGE 


SYNTHESES  TEAM  PARTICIPANTS 

•  MR  JIM  BEXFIELD 

•  MR  JERRY  GOLDSCHMIDT 

•  MR  RON  GUSTAFSON 

•  COL  JAMES  KAYS 


ISSUES  ARE  A  COMBINATION 

•  WHAT  PARTICIPANTS  ARTICULATED 

•  OUR  PERSONAL  VIEWS 


EFFECTIVENESS  ISSUE  NO.  1 


ISSUE:  JOINTNESS  IN  EFFECTIVENESS 
ANALYSIS 

DISCUSSION: 

•  CULTURE  SHOCK -“JOINT  COEAs" 

•  SINGLE  SERVICE 

•  JCS/CINSs 

•  CURRENT  MODELS  MAY  NOT  BE  ADEQUATE 

•  RELATED  TO  ISSUE  OF  CAMPAIGN  ANALYSIS 

IMPACT: 

•  POTENTIAL  FOR  CHEAPER  SOLUTIONS 

•  POTENTIAL  FOR  MORE  OPERATIONAL 
CAPABILITY 

•  BETTER  TIES  BETWEEN  ACQUISITION  PROCESS 
AND  HOW  WE  FIGHT 

•  MORE  INTER-SERVICE  ANALYST  EXCHANGES 
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EFFECTIVENESS  ISSUE  NO.  2 


ISSUE:  NEW  TECHNOLOGY  TO  IMPROVE 
MODELING 

DISCUSSION: 

•  DISTRIBUTED  INTERACTIVE  SIMULATION  (SIMNET) 

•  STANDARDS  AND  PROTOCOLS 

•  SHARING  DATA  BASES 

•  ROLE  OF  DMSO 

IMPACT: 

•  INCREASED  FLEXIBILITY 

•  REDUCED  COSTS 

•  ENHANCED  EFFICIENCY 

•  MORE  OPERATIONAL  REALISM 


EFFECTIVENESS  ISSUE  NO.  3 


ISSUE:  LINKAGE  BETWEEN  COEAs  AND  T&E 

DISCUSSION: 

•  COMPARABLE  METHODOLOGIES 

•  ESTABLISH  RELATIONSHIPS  BETWEEN  COEAs 
AND  MOEs/MOPs 

•  EXAMINE  SENSITIVITIES  TO  IDENTIFY  KEY  TEST 
VARIABLES  THAT  INFLUENCE  OPERATIONAL 
OUTCOMES 

IMPACT: 

•  MORE  CREDIBLE  &  CONSISTENT  COEAs 
AND  TESTING 

•  SYSTEMS  WHICH  BETTER  SERVE  WARFIGHTING 
NEEDS 


EFFECTIVENESS  ISSUE  NO.  4 


ISSUE:  CAMPAIGN  ANALYSIS 

DISCUSSION: 

•  WHAT  DO  WE  MEAN  BY  CAMPAIGN  ANALYSIS? 

•  WHEN  ARE  THEY  APPROPRIATE? 

•  WHO  DOES  THEM? 

•  HOW  DO  WE  ESTABLISH  CREDIBILITY? 

•  WHAT  ARE  THE  PROPER  MOEs? 

•  ARE  THE  OLD  ONES  ADEQUATE? 

•  HOW  DO  WE  CAPTURE  UNCERTAINTY? 

•  RELATION  TO  MAA? 

IMPACT: 

•  MORE  CONSISTENT  COEAs 

•  BETTER  FOCUS  ON  MILITARY  OUTCOMES 

•  BETTER  RATIONALE 


187 


188 


ANNOUNCEMENT 


M  IITARY  OPERATIONS  RESEARCH  SOCIETY 


MINI-SYMPOSIUM: 

COEA  IN  THE  ACQUISITION  PROCESS 
AND  THE  ROLE  OF  OPERATIONS  RESEARCH 
IN  PERFORMING  COEA 

DATE:  9-11  MARCH  1992 
LOCATION:  MARRIOTT  NEWPORT 
NEWPORT,  RHODE  ISLAND 

MORS  is  the  professional  association  of  military  operations  research  analysts  and  users  of  military  operations 
research  from  both  the  military  and  civilian  sector.  MORS  is  sponsored  by: 

•  The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

•  The  Director  of  Program  Resource  Appraisal,  Office  of  the  Chief  of  Naval  Operations 

•  The  Director,  Directorate  of  Programs  and  Evaluation,  Headquarters  U.S.  Air  Force 

•  The  Director  of  Force  Structure,  Resource  and  Assessment,  The  Joint  Staff 

•  The  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation) 

Under  the  Contractual  Sponsorship  of: 

•  The  Office  of  the  Chief  of  Naval  Research 


The  Proponent  for  this  mini-symposium  is  the  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation). 
Co-Proponents  are: 

The  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

The  Director  of  Program  Resource  Appraisal,  Office  of  the  Chief  of  Naval  Operations 
The  Director,  Directorate  of  Programs  and  Evaluation,  Headquarters  U.S.  Air  Force 


BACKGROUND 

One  of  the  Secretary  of  Defense’s  major 
initiatives  was  to  streamline  the  acquisition  process. 
Accordingly,  the  Defense  Acquisition  process  was 
realigned  in  the  new  DoD  5000  Series  Regulations  to 
establish  a  disciplined  management  approach  for 
acquiring  systems  and  materiel  that  satisfy  the 
operational  user’s  needs.  During  a  time  of  rapid 
changes  in  the  world  and  constrained  resources,  it  is 
of  the  utmost  importance  that  military  decision 
makers  within  the  acquisition  process  are  supported 


by  pertinent  and  timely  analysis.  These  new 
regulations  now  contain  general  guidelines  and  place 
greater  emphasis  on  the  Cost  and  Operational 
Effectiveness  Analyses  (COEA)  which  help  decision 
makers  select  the  best  systems  from  among 
alternatives.  The  COEA  is  an  important  document  in 
the  acquisition  decision  making  process  to  support 
both  effectiveness  and  affordability  assessments  and 
program  milestone  decisions. 
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In  an  attempt  to  enhance  and  upgrade  the  level  of 
analytical  expectation,  OASD(PA&E)  developed  and 
published  COEA  guidelines  which  were  incorporated 
into  the  new  DoD  5000  Series  Regulations.  Although 
the  requirement  for  COEA  previously  existed,  the 
regulations  now  contain  a  clearer  framework  to 
conduct  COEA.  Recognizing  the  importance  of 
COEA,  PA&E  determined  that  a  series  of 
workshops,  explaining  the  relationship  of  the  COEA 
to  the  acquisition  decision  making  process,  was 
appropriate  and  timely.  PA&E  viewed  the 

workshops  as  a  way  to  start  improving  the  overall 
analysis  support  for  acquisition  decisions.  By 
bringing  all  the  services  and  acquisition  activities 
together  to  discuss  COEA,  common  problems  could 
be  addressed  and  resolved.  PA&E  also  felt  that  a 
workshop  would  help  clarify  what  is  expected  in  a 
COEA  and  would  also  help  establish  a  more 
collaborative  framework  within  the  DoD  structure  for 
dealing  with  COEA  requirements  in  the  future. 

A  workshop  for  senior  officials  and  analysts  from 
DoD  and  the  military  departments  was  held  on  3 
April  1991  at  the  Defense  Systems  Management 
College,  Fort  Belvoir,  Virginia.  Seventy-two  general 
officers,  senior  executive  service  civilians  and  others 
attended.  The  workshop  was  opened  by  Dr.  David 
Chu,  ASD(PA&E)  who  discussed  the  problems  with 
current  COEA.  Short  talks  by  subject  matter  experts 
were  then  given,  each  followed  by  a  brief  discussion 
period.  At  the  end  of  the  day  each  service  presented 
its  unique  problems  and  perceptions  of  the  value  of 
the  workshop. 

Three  "limited  attendance"  action  officer 
workshops  were  held  in  May  1991,  with  the  MITRE 
Corporation  serving  as  host.  Each  workshop  lasted 
two  days.  Dr.  Chu  opened  each  workshop  by  again 
stressing  the  need  for  better  analysis  as  he  had  done 
at  the  Senior  Officer  Workshop.  Workshop  topics 
were  presented,  again  by  subject  matter  experts. 

A  tutorial  concerning  the  PA&E  workshops  was 
presented  at  the  recent  Annual  MORS  Symposium. 
The  tutorial  focused  on  the  conduct  of  the  workshops, 
on  issues  generated  from  the  workshops  and  on 
trends  or  perceptions  that  came  from  conducting  and 
observing  the  workshops. 

In  all  the  workshops  and  tutorials,  interest  from 
all  segments  of  DoD  has  been  much  higher  than 
anticipated.  Numerous  requests  to  hold  additional 
workshops  for  DoD  as  well  as  contractor  personnel 
have  been  received. 


OBJECTIVE 

The  overall  objective  of  the  mini-symposium  is  to 
explore  the  COEA  process  and  identify  how  the 
application  of  operations  research  principles  and 
techniques  might  improve  the  information  available  to 
decision  makers.  In  particular  the  goals  are  to: 

•  understand  the  role  of  COEA  in  the  DoD 
5000  Series  Acquisition  Regulations 

•  define  the  role  of  operations  research  in 
the  COEA  process 

•  identify  challenges  and  issues  associated 
with  conducting  COEA  and  identifying 
those  challenges  that  could  best  be 
addressed  by  operations  research 

•  develop  possible  solutions  or  identify 
appropriate  research  areas  to  those 
operations  research  problems  and 
issues 

•  improve  the  collaborative  framework  for 
dealing  with  COEA  requirements 

SCOPE 

The  mini-symposium  will  cover  a  three  day 
period.  Dr.  David  Chu  will  be  the  keynote  speaker. 
Another  senior  DoD  or  Congressional  official  will 
also  make  a  presentation  on  a  significant  analysis 
issue  of  the  day.  The  mini-symposium  will  provide 
an  opportunity  for  MORS  sponsors  and  members  to 
discuss  their  views  concerning  both  the  COEA 
process  and  the  role  of  operations  research  in 
performing  COEA.  After  the  last  presentation  a 
panel  of  senior  DoD  personnel  will  discuss  the  future 
direction  of  COEA.  Subject  matter  experts  will  be 
invited  by  the  chairman  to  present  a  series  of  papers 
on  the  following  topics: 

(1)  Milestone  Decision  Criteria 

(2)  Linkage  of  Performance  Effectiveness 

(3)  Threat  and  Scenarios 

(4)  Operational  Effectiveness  Methodologies 

(5)  Costing  Methodologies 

(6)  Cost  and  Effectiveness  Integration 

(7)  Modeling  and  Tools 

(81  Managing  COEA 

(9)  COEA  Guidelines 
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Each  presentation  will  last  twenty-five  minutes. 
After  every  three  presentations  the  audience  will  be 
divided  into  three  sections.  Each  section  will  then 
enjoy  a  twenty-five  minute  Q&A  session  with  each 
presenter.  The  audience  will  be  divided  into  six 
working  groups.  Symposium  attendees  will  be  pre¬ 
assigned  to  a  working  group  to  ensure  that  each 
working  group  contains  representatives  from  the 
various  segments  of  the  operations  research 
community.  The  working  groups  will  focus  and 


report  on 

the  following  six  topics: 

(D 

Test  and  Evaluation 

(2) 

Threat  Assessments 

(3) 

Effectiveness  Methodologies 

(4) 

Costing  Methodologies 

(5) 

COEA  Management 

(6) 

Future  Directions 

The  mini-symposium  chair  will  select  a  chair  for 
each  working  group.  The  group  chairs  should 
provide  a  breadth  of  experience  and  collectively 
represent  the  various  sponsors  and  organizations  that 
either  conduct  COEA  or  use  COEA  in  decision 
making.  The  working  group  chairs  are  responsible 
for  guiding  their  working  group  discussions,  and 
insuring  their  group  develops  a  set  of  issues  and 
problems,  potential  solutions  to  those  issues  and 
problems,  and  a  presentation  of  the  working  group 
results  to  the  entire  body.  A  panel  consisting  of  six 
chairs  (one  from  each  working  group)  will  be  held  in 
a  general  assembly  to  discuss  each  group’s  issues  and 
perceptions. 

A  synthesis  group  will  observe  the  mini¬ 
symposium’s  general  and  working  group  sessions  to 
identify  issues  that  merit  investigation  in  future 
MORS  special  meetings.  The  synthesis  group 
members  will  have  prior  knowledge  of  the  role  of 
COEA  in  the  acquisition  process. 

PARTICIPATION 

Attendance  will  be  limited  to  a  maximum  of  200 
attendees.  The  goal  is  to  get  a  mix  of  people  with 
various  levels  of  COEA  experience  but  more 
importantly  each  attendee  should  be  currently 
involved  with  COEA  and  should  come  prepared  to 
participate  in  an  active  role  as  recorder,  moderator, 
or  discussion  leader.  The  symposium,  similar  to  the 


PA&E  workshops,  will  be  held  at  the  unclassified 
level. 

LUNCHEON  SPEAKER 

A  luncheon  will  be  held  at  the  Marriott  Hotel  on 
Tuesday,  10  March  1992.  A  presentation  will  be 
made  by  a  Member  of  Congress  on  topics  relevant  to 
the  symposium  theme.  The  luncheon  cost  is  included 
in  the  registration  fee. 

COMMITTEE 

General  Chair: 

Dr.  A1  Diaz 

Naval  War  College 

Newport,  RI  08241 

(401)  841-3850  FAX  (401)  841-3804 

Deputy  Chairs: 

LtCol  Dewey  Tucker 
OASD(PA&E) 

The  Pentagon,  Room  2B256 
Washington,  D.C.  20301-1800 
(703)  697-7085  FAX  (703)  693-5707 

Mr.  Edward  F.  Smith,  Jr. 

Institute  for  Defense  Analyses 
1801  N.  Beauregard  Street 
Alexandria,  VA  22311 
(703)  845-6938  FAX  (703)  845-6911 

Other  Members: 

Mr.  John  Riente 
Technical  Advisor 

Office  of  the  Deputy  Chief  of  Staff  for  Operations 
and  Plans 

HQ  Department  of  the  Army 
Washington,  DC  20310-0410 
(703)  697-4113  FAX  (703)  694-9044 

Mr.  Clayton  J.  Thomas,  FS 
Chief  Scientist 

Air  Force  Studies  and  Analyses  Agency 
HQ  USAF/SAN 
Washington,  DC  20330-5420 
(703)  697-4300  FAX  697-3441 

CDR  Stan  H.  Hlavka 

Office  of  the  Chief  of  Naval  Operations  (OP815D) 

Department  of  the  Navy 

Washington,  DC  20350-2000 

(703)  697-8078  FAX  (703)  695-6903 
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Mr.  Don  Mixon 

MITRE  Corporation 

7525  Colshire  Dr.,  MS-W966 

McLean,  VA  22102 

(703)  883-6599  FAX  (703)  883-5963 

LtCol  Don  Bourdon,  USAF 
Naval  War  College 
Newport,  RI  02841 
(401)  841-3892  FAX  (401)  841-3804 

Mr.  Richard  I.  Wiles 

Executive  Director 

Military  Operations  Research  Society 

101  S.  Whiting  Street,  Suite  202 

Alexandria,  VA  22304-3483 

(703)  751-7290  FAX  (703)  751-8171 

Ms.  Natalie  S.  Addison 
Associate  Executive  Director 
Military  Operations  Research  Society 
101  S.  Whiting  Street,  Suite  202 
Alexandria,  VA  22304-3483 
(703)  751-7290  FAX  (703)  751-8171 

MORS  Advisor: 

Dr.  Bruce  MacDonald 

MITRE  Corporation 

7525  Colshire  Dr.,  MS-W272 

McLean,  VA  22102 

(703)  883-7855  FAX  (703)  883-6478 

PRODUCTS 

The  presenters  will  be  asked  to  provide  a  paper 
on  their  topic  to  form  the  basis  for  a  proceedings. 
Dr.  Chu’s  talk  and  the  reports  from  each  working 
group  will  also  be  included  as  part  of  the 
proceedings.  Additionally  the  synthesis  group’s 
report  will  be  included  as  part  of  the  proceedings  and 
will  serve  as  the  basis  for  planning  future  MORS 
special  meetings. 

In  addition,  the  achievements  of  the  meeting  will 
be  reported  in  a  presentation  to  the  General  Session 
of  the  60th  MORSS  and  in  a  PHALANX  article. 

LOGISTICS 

The  Marriott  Hotel  in  Newport,  Rhode  Island  will 
be  the  site  for  this  mini-symposium.  The  daily  rate 
is  $93  including  tax.  Call  (401)  849-1000  for 
reservations  and  mention  that  you  are  a  participant  in 


the  MORS  Mini-Symposium.  Reservations  must  be 
made  by  16  February  1992.  The  address  is:  Newport 
Marriott,  25  America’s  Cup  Avenue,  Newport, 
Rhode  Island  02840.  The  closest  Airport  is  T.F. 
Green  in  Providence,  Rhode  Island. 

FEES 

The  Registration  Fee  is  $180.00  for  Federal 
Government  and  $330.00  for  other  personnel.  The 
Fee  includes  $20.00  for  a  luncheon  (with  speaker)  on 
Tuesday  and  $10.00  for  a  working  lunch  on 
Wednesday. 

CAVEATS 

The  Military  Operations  Research  Society  does 
not  make  nor  advocate  official  policy. 

Matters  discussed  or  statements  made  during  the 
symposium  are  the  sole  responsibility  of  the 
participants  involved. 

All  attendees  and  participants  are  expected  to 
submit  requisite  attendance  forms  and  to  pay  the 
normal  registration  fees  unless  specifically  waived  by 
the  MORS  President.  There  is  no  waiver  or  discount 
for  short-period  attendance  or  participation. 

The  Society  retains  all  rights  regarding  final 
decision  on  the  content  of  the  Mini-Symposium 
Report. 


Approved: 


iU 


Jerome  X.  Goldschmidt 

Contracting  Officer’s  Technical  Representative 
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MORS  CQEA  Symposium 

Meeting  Agenda  for  March  9-11.  1992 


MONDAY.  9  MARCH  1992 

0800-1000  Registration 

1000-1015  Opening  Remarks 

A1  Diaz,  Chairman 

1015-1030  MORS  Welcome 

Vern  Bettencourt,  MORS  President 

1030-1045  Host  Welcome 

(Naval  War  College  Representative) 

1045-1145  Keynote 

David  Chu,  Assistant  Secretary  of  Defense  (Program  Analysis  and  Evaluation) 

1145-1245  Luncheon 

1245-1455  GENERAL  SESSION 

(25  min)  Topic  #1:  Milestone  Decision  Criteria 
(25  min)  Topic  #2:  Threat  and  Scenarios 

1335-1430  Hon.  Ron  Machtley 

(25  min)  Topic  #3:  Modelling  and  Tools 

1455-1515  Break 


1515-1630  QUESTION  &  ANSWER  SESSIONS 

(25  min)  WGs  1&2:  Topic  #1  WGs  3&4:  Topic  #2 

(25  min)  WGs  1&2:  Topic  #2  WGs  3&4:  Topic  #3 


(25  min)  WGs  1&2:  Topic  #3  WGs  3&4:  Topic  #1 


WGs  5&6:  Topic  #3 
WGs  5&6:  Topic  #1 
WGs  5&6:  Topic  #2 


1630-1730  WORKING  GROUP  SESSION 

1800-1900  Hors  D’oeuvres  at  Marriott  Hotel 


TUESDAY,  10  MARCH  1992 

0800-0915  GENERAL  SESSION 

(25  min)  Topic  #4:  Operational  Effectiveness  Methodologies 

(25  min)  Topic  #5:  Costing  Methodologies 

(25  min)  Topic  #6:  Cost  and  Effectiveness  Integration 


0915-1030  QUESTION  &  ANSWER  SESSIONS 

(25  min)  WGs  1&2:  Topic  #4  WGs  3&4:  Topic  #5 

(25  min)  WGs  1&2:  Topic  #5  WGs  3&4:  Topic  #6 


(25  min)  WGs  1&2:  Topic  #6  WGs  3&4:  Topic  #4 
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WGs  5&6:  Topic  #6 
WGs  5&6:  Topic  #4 
WGs  5&6:  Topic  #5 


1030-1045  Break 

1045-1200  GENERAL  SESSION 

(25  min)  Topic  #7:  Managing  COEA 

(25  min)  Topic  #8:  Linkage  of  Performance  Effectiveness 

(25  min)  Topic  #9:  COEA  Guidelines 

1200-1330  Lunch 


1330-1445  QUESTION  &  ANSWER  SESSIONS 

(25  min)  WGs  1&2:  Topic  #7  WGs  3&4:  Topic  #8 

(25  min)  WGs  1&2:  Topic  #8  WGs  3&4:  Topic  #9 


(25  min)  WGs  1&2:  Topic  #9  WGs  3&4:  Topic  #7 


1445-1500  Break 


1500-1700  Panel  on  Future  Direction 


1700-2100  WORKING  GROUP  SESSIONS 


WGs  5&6:  Topic  #9 
WGs  5&6:  Topic  #7 
WGs  5&6:  Topic  #8 


WEDNESDAY,  11  MARCH  1992 

0800-1200  WORKING  GROUP  SESSIONS 

Working  Groups  1  through  6:  Topic  -  Define  and  report  on  the  role  of  operations  research  in  the 
COEA  process;  issues,  problems,  and  research  areas,  and  the  future  directions  for  military 
operations  research  analysis  to  support  COEAs. 


1200-1300 


Working  Lunch  for  Working  Groups 


1300-1400 


GENERAL  SESSION  (REPORTS  TO  THE  ASSEMBLY  BY  WORKING  GROUP  CHAIRMEN) 


(20  min  each)  Working  Groups  1  through  3 
1400-1415  Break 

1415-1515  GENERAL  SESSION  (REPORTS  TO  THE  ASSEMBLY  BY  WORKING  GROUP  CHAIRMEN) 

(20  min  each)  Working  Groups  4  through  6 


1515-1545 


Synthesis  Group  Feedback 


1545-1600  WRAP-UP/CLOSING 
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INVITED  SPEAKERS 


SPEAKER 
Dr  Bill  Lese  (OASD(PA&E)) 

LtCol  Gary  Fauss  (DIA) 

Mr  Sam  Gardiner 

Mr  Tony  Brinkley  (Teledyne  Brown) 

Dr  Dan  Nussbaum  (CNCA) 

Dr  Pat  Cassady  (TRAC-WSMR) 

LtCol  Jim  Feigley  (USMC) 

Bill  Lese  (PA&E) 

Ernie  Seglie  (DOTE) 

Dick  Ledesma  (DDR&E) 

Dr  A1  Diaz  (OASD(PA&E)) 


TOPIC 

Milestone  Decision  Criteria 
Threats  and  Scenarios 
Modeling  and  Tools 

Operational  Effectiveness  Methodologies 

Cost  Methodologies 

Cost  and  Effectiveness  Integration 

Managing  COEA 

Linkage  of  Testing  and  COEAs 

COEA  GUIDELINES 
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WORKING  GROUP  LEADERS 


Working  Group 

Costing  Methodologies 

COEA  Management 

Effectiveness 

Methodologies 

Future  Directions 

Threats  &  Scenarios 

Test  &  Evaluation 

Synthesis  Group 


Chairman 

Dr  Dan  Nussbaum  (CNCA) 
LtCol  Don  Bourdon  (NWC) 

Mr  John  Friel  (RAND) 

Mr  Mike  Bauman  (TRAC) 

Col  Charlie  Cox  (DIA) 

Mr  John  Gehrig(ODU S A(OR) 
Mr  Clayton  Thomas  (USAFSA) 


Moderator 

LtCol  Ray  Baker,  USAF  (NWC) 
Cdr  Ed  Berry,  USN  (NWC) 

Cdr  Indy  Crowley, USN  (NWC) 

Cdr  Scott  Ensminger.USN  (NWC) 
Cdr  Chuck  Sipe,  USN  (NWC) 
LtCol  John  Sergeant,  USMC 
Professor  Steve  Fought  (NWC) 
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March  10,  1992 


Robert  D  Aaron 

US  Army  Operational  Test  &  Eval  Cmd 

ATTN:  CSTE-ECC-Z 

4501  Ford  Ave,  //920 

Alexandria, VA  22302-1458 

OFFICE  PHONE:  (703) -756-2306 

WG:  Test  &  Evaluation 

Paul  E.  Albers 

The  MITRE  Corporation 

1745  Jefferson  Davis  Hwy,  Suite  401 

Arlington, VA  22202 

OFFICE  PHONE:  ( 703) -271-8009 

WG:  Effectiveness  Methodologies 


Steven  E  Anderson 
Naval  Surface  Warfare  Center 
Systems  Accuracy  Branch  G12 
Trailer  5 

Dahlgren,VA  22448-5000 

OFFICE  PHONE:  ( 703) -663-8538  DSN:  249-8538 
WG:  Threat  Assessment 

DR  Jerry  B.  Arnett 
ASD/XRX 

Wright-Patterson  AFB , OH  45433-6503 
OFFICE  PHONE:  (513)-255-6261  DSN:  785-6261 
WG:  Future  Directions 


LtCol  Raymond  G  Baker 
Naval  War  College 
Code  BD3 

Newport, RI  02841 

OFFICE  PHONE:  (401) -841-3496  DSN:  948-3496 
WG:  Costing  Methodologies 


William  D.  Barr 
US  Army  MISMA 

Suite  808,  Crystal  Square  2 
1725  Jefferson  Davis  Hwy 
Arlington, VA  22202 
OFFICE  PHONE:  (703)-746-8073 
WG:  Future  Directions 


Natalie  S  Addison 

Military  Operations  Research  Society 
101  S  Whiting  Street 
Suite  202 

Alexandria, VA  22304-3483 
OFFICE  PHONE:  (703) -751-7290 
WG:  Organizing  Committee 

RADM  R.  C.  Allen 
OCNO  OP-81 

Department  of  the  Navy 
The  Pentagon,  Room  4A530 
Washington, DC  20350-2000 

OFFICE  PHONE:  (703) -697-0831  DSN:  227-0831 
WG:  Speaker  or  Panelist 

H.  Lindsey  Arison  III 
SAF/FMBM 

Office  of  the  Asst,  for  Special  Programs 

The  Pentagon,  Room  5C132 

Washington, DC  20330 

OFFICE  PHONE:  (703) -614-1283 

WG:  Costing  Methodologies 

DR  Francis  E  Baker 

Naval  Surface  Warfare  Center 

Code  N10 

10901  New  Hampshire  Ave 

Silver  Spring, MD  20903-5000 

OFFICE  PHONE:  ( 301 ) -394-1269  DSN:  290-1269 

WG:  Costing  Methodologies 

DR  Donald  R  Barr 
Naval  Postgraduate  School 
Code  MA/BA 
Monterey, CA  93943 

OFFICE  PHONE:  (408) -646-2763  DSN:  878-2763 
WG:  Test  &  Evaluation 


Michael  F  Bauman 
HQ  TRADOC  Analysis  Command 
ATTN:  ATRC-ZD 

Fort  Leavenworth,KS  66027-5200 

OFFICE  PHONE:  (913)-684-4689  DSN:  552-468- 

WG:  Future  Directions 
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Larry  J  Beasley 
ASD/ENSS 

Wright-Patterson  AFB.OH  45433-6503 
OFFICE  PHONE:  (513 ) -255-5093  DSN:  785-5093 
WG:  Effectiveness  Methodologies 


Frederick  C.  Belen 
ASN  (RDA)  EEP 
The  Pentagon,  Room  5E731 
Washington, DC  20350-1000 
OFFICE  PHONE:  ( 703 ) -614-4794 
WG:  Future  Directions 


Vernon  M  Bettencourt  Jr 
The  MITRE  Corporation 
W  448 

7525  Colshire  Drive 
McLean, VA  22102-3481 
OFFICE  PHONE:  (703) -883-7364 
WG:  Synthesis  Group 

Thomas  A  Blanco 

Navy  Personnel  Researh  &  Development  Ctr 
Code  112 

San  Diego, CA  92152-6800 
OFFICE  PHONE:  (619) -553-8044 
WG:  Effectiveness  Methodologies 


DR  Gregory  J  Born 
RAND 

1700  Main  Street 
Santa  Monica, CA  90407-2138 
OFFICE  PHONE:  (310)-393-0411 
WG:  Effectiveness  Methodologies 


LtCol  Donald  J  Bourdon 
Naval  War  College 
NSDM  Dept 
Newport, RI  02841 

OFFICE  PHONE:  (401 ) -841-3496  DSN:  948-3496 
WG :  COEA  Management 


James  R  Behne 

TRADOC  Analysis  Command-Lee 

Attn:  ATRC-LS 

Fort  Lee,VA  23801-6140 

OFFICE  PHONE:  (804)-734-4971  DSN:  687-4971 
WG:  COEA  Management 


CDR  Edmund  W  Berry 
Naval  War  College 
NSDM  Dept 
Newport, RI  02840 

OFFICE  PHONE:  (401)-841-3892  DSN:  948-3892 
WG :  COEA  Management 


James  N  Bexfield 
Institute  for  Defense  Analyses 
1801  N.  Beauregard 
Alexandria, VA  22311 

OFFICE  PHONE:  (703) -845-2107  DSN:  289-1829 
WG:  Synthesis  Group 


DR  Stephen  A  Book 
The  Aerospace  Corporation 
P.0.  Box  92957 
M/S  M5-021 

Los  Angeles, CA  90009-2957 
OFFICE  PHONE:  (310) -336-8655 
WG:  Costing  Methodologies 

Paul  J  Boucher 

US  Army  Communications  Electronics  Cmd 

ATTN:  AMSEL-CP-CA 

Fort  Monmouth, NJ  07703-5034 

OFFICE  PHONE:  (908)-544-2711 

WG:  Costing  Methodologies 


COL  Ronald  S  Bowen 
SAF/FMC 

Pentagon,  Room  4D159 
Washington, DC  20330-1000 
OFFICE  PHONE:  (703) -697-5312 
WG:  Costing  Methodologies 
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.  .  (cont.) 

LCDR  Benjamin  F  Breaux 

OCNO  (OP-81),  Department  of  the  Navy 

Washington, DC  20350-2000 

OFFICE  PHONE:  (703)-697-8078  DSN:  227-8078 

WG :  COEA  Management 


Attendance  at  MORS  Mini-Symposium:  COEA  . 
LtCol  David  G  Boyd 

US  Army  Operational  Test  &  Eval  Command 

4501  Ford  Avenue 

Alexandria ,VA  22302 

OFFICE  PHONE:  (703)-756-1817 

WG:  Test  &  Evaluation 


Timothy  E  Bright 
OASD/PA&E/GPP/LFD 
Pentagon,  Room  2B256 
Washington, DC  20301 

OFFICE  PHONE:  (703) -697-6408  DSN:  221-6408 
WG:  Effectiveness  Methodologies 


Aaron  W  Burstein 

Naval  Air  Development  Center 

Aircraft  Division 

Code  302B 

Warminster, PA  18974 

OFFICE  PHONE:  (215)-441-2267  DSN:  441-2267 
WG:  Costing  Methodologies 

DR  Monti  D  Callero 
The  RAND  Corporation 
1700  Main  St 

Santa  Monica, CA  90406-2138 
OFFICE  PHONE:  ( 310) -393-0411 
WG :  COEA  Management 


LtCol  James  G  Casler 
HQ,  US  Marine  Corps 
Requirements  and  Programs 
Washington, DC  20380-0001 
OFFICE  PHONE:  (703) -614-1503 
WG:  Test  &  Evaluation 


LtCol  Frederick  W  Chapman  Jr 
HQ  USAF/XORJ 
Pentagon,  Room  BF938A 
Washington, DC  20330-5057 
OFFICE  PHONE:  (703) -695-7107 
WG:  COEA  Management 


William  A  Brinkley 
Teledyne  Brown  Engineering 
PO  Box  070007 
300  Sparkman  Dr,  MS  16 
Huntsville, AL  35807-7007 
OFFICE  PHONE:  ( 205) -726-5857 
WG:  Effectiveness  Methodologies 

Capt  Jan  E  Caffey 
US  Army  Logistics  College 
ATTN:  ATSZ-LSS 
Fort  Lee ,VA  23801-6050 

OFFICE  PHONE:  (804)-734-5431  DSN:  687-5431 
WG:  Costing  Methodologies 


Daniel  L  Carbo 
Naval  Air  Warfare  Center 
Aircraft  Div-Warminster 
Warminster, PA  18974-5000 

OFFICE  PHONE:  (215) -441-7006  DSN:  441-7006 
WG:  Costing  Methodologies 


DR  Patrick  D  Cassady 

TRAC-WSMR 

Attn:  ATRC-WAC 

White  Sands  Missile  Range, NM  88002 
OFFICE  PHONE:  (505) -678-4390  DSN:  258-4390 
WG:  Future  Directions 


LtCol  Robert  M.  Chapman 
HQ  TAC/XP-JSG 
HQ  TAX/XP 

Langley  AFB ,VA  23665-5520 

OFFICE  PHONE:  (804)-764-5757  DSN:  574-575* 
WG:  Costing  Methodologies 
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Kuei-Lin  Chen 
OASD ( PA&E ) 

Pentagon,  Room  2D312 

Washington, DC  20301-1800 

OFFICE  PHONE:  ( 703) -697-0063  DSN:  227-0063 

WG:  Threat  Assessment 


Shelia  T.  Clapp 

Consultant 

69  Tenney  Road 

Westford,MA  01886 

OFFICE  PHONE:  ( 508 ) -692-4292 

WG:  Costing  Methodologies 


Linda  Coblentz 

US  Army  Concepts  Analysis  Agency 
Attn:  CSCA-ESR 
8120  Woodmont  Ave 
Bethesda,MD  20814 

OFFICE  PHONE:  (301)-295-1546  DSN:  295-1546 
WG:  Costing  Methodologies 

William  F  Connor 

Systems  Planning  and  Analysis,  Inc 
5111  Leesburg  Pike,  #200 
Falls  Church, VA  22041 
OFFICE  PHONE:  (703) -931-3500 
WG:  Effectiveness  Methodologies 


Terence  J .  Cooney 
Veda,  Inc 

5200  Springfield  Pike,  #200 
Beavercreek, OH  45431-1255 
OFFICE  PHONE:  (513) -253-4770 
WG:  Future  Directions 


COL  Charles  0  Cox 

Defense  Intelligence  Agency 

Attn:  ODT-AS 

Washington, DC  20340-6132 

OFFICE  PHONE:  (202) -373-8026  DSN:  243-8026 

WG:  Threat  Assessment 


DR  David  SC  Chu 
ASD  (PA&E) 

The  Pentagon 
Room  3E836 

Washington, DC  20301-1800 

OFFICE  PHONE:  (703)-695-0971  DSN:  225-0971 
WG:  Speaker  or  Panelist 

Nancy  L  Clements 

AFSC 

ASD/ARM 

Wright-Patterson  AFB ,0H  45433-6503 
OFFICE  PHONE:  (513)-255-6261  DSN:  785-6261 
WG:  Costing  Methodologies 


DR  Gary  C  Comfort 
Institute  for  Defense  Analyses 
1801  N.  Beauregard  Street 
Alexandria, VA  22311-1772 
OFFICE  PHONE:  (703) -845-2245 
WG:  Test  &  Evaluation 


Joseph  M  Conrad 
SVERDRUP  Technology  Inc. 

TEAS  Group,  Bldg  260 

PO  Box  1935 

Eglin  AFB , FL  32542 

OFFICE  PHONE:  (904) -678-2001 

WG:  Effectiveness  Methodologies 

William  J  Cooper 
US  Army  MTMC 
Attn:  MTT-OA 
PO  Box  6276 

Newport  News ,VA  23606-0276 
OFFICE  PHONE:  ( 804) -599-1650 
WG:  Future  Directions 

LtCol  James  W  Crawford 
HQ  USSOCO 

US  Speical  Operations  Command  (J8) 

MacDill  AFB , FL  33619 

OFFICE  PHONE:  (813)-830-4294  DSN:  968-429- 
WG:  COEA  Management 
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Indy  Crowley 
Naval  War  College 
NSDM  Dept 

Newport, RI  02841-5010 

OFFICE  PHONE:  ( 401 ) -841-3892  DSN:  948-3892 
WG:  Effectiveness  Methodologies 


DR  Alfonso  A  Diaz 
OASD ( PA&E ) 

General  Purpose  Programs 
The  Pentagon,  Room  2B-256 
Washington, DC  20301 
OFFICE  PHONE:  ( 703) -697-7768 
WG:  Organizing  Committee 

LtCol  Robert  C  Doheny 
OASD  (SO/LIC)  -  Acquisition 
Pentagon,  RM  1A674B 
Washington, DC  20301 
OFFICE  PHONE:  ( 703) -693-5222 
WG :  COEA  Management 


DR  Henry  C  Dubin 

US  Army  Operational  Test  &  Eval  Comd 
Park  Center  IV 
4501  Ford  Avenue 
Alexandria, VA  22302-1458 

OFFICE  PHONE:  (703)-756-2367  DSN:  289-2367 
WG:  Test  &  Evaluation 

Maria  M  Enriquez 

US  Army  Air  Defense  Artillery  School 
Directorate  of  Combat  Development 
Attn:  ATSA-CDC-S 
Fort  Bliss, TX  79916 

OFFICE  PHONE:  (915)-568-1933  DSN:  978-3157 
WG:  Threat  Assessment 

COL  Joseph  L  Faix 

Office  of  Aerospace  Studies 

OAS(AFSC) 

Kirtland  AFB,NM  87117-6008 

OFFICE  PHONE:  (505)-846-8322  DSN:  246-8322 

WG :  COEA  Management 


Joseph  P.  Dean 
Tecolote  Research  Inc. 

54  Middlesex  Turnpike 
Bedford, MA  01730 
OFFICE  PHONE:  (617)-275-3014 
WG:  Future  Directions 


Dominic  P  DiOrio 
NUWC,  NPT  Division 
Newport, RI  02841 
OFFICE  PHONE:  (401 )-841-3027 
WG:  COEA  Management 


Hubert  W  Drake 
AS I  Systems  International 
825  N  Downs,  Suite  D 
Ridgecrest, CA  93555 
OFFICE  PHONE:  (619) -375-1442 
WG:  Threat  Assessment 


MAJ  Daniel  E.  Durham 
Ballistic  Missile  Office/MYSP 
Norton  AFB,CA  92409-6468 

OFFICE  PHONE:  (714)-382-2592  DSN:  876-2592 
WG:  COEA  Management 


CDR  D  S  Ensminger 
Naval  War  College 
NSDM  Dept 

Newport, RI  02841-5010 

OFFICE  PHONE:  (701)-841-3892  DSN:  948-3892 
WG:  Future  Directions 


LtCol  Gary  E  Fauss 
Defense  Intelligence  Agency 
Attn:  ODT-AS 
Washington, DC  20340-6132 

OFFICE  PHONE:  (202) -373-8370  DSN:  243-837C 
WG:  Threat  Assessment 
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LtCol  Jim  Feigley 

HQMC 

DPM-AAA 

Washington, DC  20380-0001 
OFFICE  PHONE:  (703)-696-1119 
WG :  COEA  Management 


MAJ  Matthew  A.  Pinion 

MAGTF  Warfighting  Center 

Studies  and  Analysis  Branch  (WF13) 

Marine  Corps  Combat  Development  Command 
Quant ico,VA  22134 

OFFICE  PHONE:  (703) -640-5989  DSN:  278-5989 
WG:  COEA  Management 

Teresa  A  Forrest 
Alii ant  Techsys terns 
5901  Lincoln  Dr 
Edina, MN  55436 

OFFICE  PHONE:  (612) -939-3064 
WG:  Effectiveness  Methodologies 


PROF  Stephen  0  Fought 
Naval  War  College 
Newport, RI  02841 

OFFICE  PHONE:  (401)-841-3496  DSN:  948-3496 
WG:  Synthesis  Group 


Roger  A  Francis 
Horizons  Technology  Inc 
700  Technology  Park  Drive 
Billerca.MA  01821-4196 
OFFICE  PHONE:  (508) -663-6600 
WG:  Effectiveness  Methodologies 


Aimer  B.  Galloway 

Naval  Air  Warfare  Center  Weapons  Div 
Weapons  Division 
Code  308 

Ridgecrest, CA  93555-6001 

OFFICE  PHONE:  (619)-939-3811  DSN:  437-2701 
WG:  Threat  Assessment 


Dennis  A  Fenn 

McDonnell  Aircraft  Co 

MC  0642333 

PO  Box  516 

St.  Louis, MO  63166 

OFFICE  PHONE:  (314)-777-5224 

WG:  Costing  Methodologies 

LtCol  Michael  J  Foley 

HQ  TAC/XP-JSG 

Joint  Studies  Group 

Langley  AFB.VA  23665-5520 

OFFICE  PHONE:  (804) -764-2064  DSN:  574-206A 

WG:  Costing  Methodologies 


Stephen  Foster 

Naval  Air  Warfare  Center,  Weapons  Div 
Code  3081 

Ridgecrest, CA  93555-6001 

OFFICE  PHONE:  (619)-939-3081  DSN:  437-3081 

WG:  Costing  Methodologies 


DR  Bruce  W  Fowler 

US  Army  Missile  Command 

Attn:  AMSMI-RD-AC 

Redstone  Arsenal, AL  35898-5242 

OFFICE  PHONE:  (205)-876-8173  DSN:  746-8173 

WG:  Future  Directions 


DR  John  A  Friel 
The  RAND  Corporation 
1700  Main  Street 
P  0  Box  2138 

Santa  Monica, CA  90406-2138 
OFFICE  PHONE:  (310) -393-0411 
WG:  Effectiveness  Methodologies 

Samuel  B  Gardiner 

Consultant 

1308  21st  St 

Arlington, VA  22202 

OFFICE  PHONE:  (703) -979-3685 

WG:  Speaker  or  Panelist 
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Maura  D.  Garvey 

ARINC  Research  Corporation 

70  Westview  Street 

Lexington, MA  02173 

OFFICE  PHONE:  (617) -860-4553 

WG:  Effectiveness  Methodologies 


John  E.  Gibson  Jr. 

Northrop  Coporation 
MS  N400/GG 

8900  E.  Washington  Blvd 
Pico  Rivera, CA  90660-3737 
OFFICE  PHONE:  ( 310) -948-7780 
WG :  COEA  Management 

Russell  E  Gingras 

Johns  Hopkins  University/APL 

Concept  Evaluation  Group 

Johns  Hopkins  Road 

Laurel, MD  20723 

OFFICE  PHONE:  (301 ) -953-5066 

WG:  Effectiveness  Methodologies 

Edwin  M  Goldberg 
US  Army,  CEC0M 
Attn:  AMSEL-PE-SA 
Fort  Monmouth, NJ  07703 

OFFICE  PHONE:  (908) -532-4684  DSN:  992-4684 
WG:  Future  Directions 


Jerome  X  Goldschmidt 

OCNO  (OP-816) ,  Dept  of  the  Navy 

Pentagon,  Room  4A510 

Washington, DC  20350-2000 

OFFICE  PHONE:  (703) -697-8051  DSN:  227-8051 

WG:  Synthesis  Group 


Stephen  R  Gordon 
HQ  MAC/XPY 

Scott  AFB, IL  62225-5001 
OFFICE  PHONE:  (618) -256-3450 
WG:  Effectiveness  Methodologies 


John  F.  Gehrig 

HQDA 

DACS-TE 

The  Pentagon,  Room  3C567 
Washington, DC  20310-0102 
OFFICE  PHONE:  ( 703) -695-8995 
WG:  Test  &  Evaluation 

David  J  Gill 

Rockwell  International 

Tactical  Systems  Division 

1800  Satellite  Blvd 

Duluth, GA  30136 

OFFICE  PHONE:  ( 404) -497-5196 

WG:  Effectiveness  Methodologies 

Jonathan  Given 
US  Army  Natic  RD&E  Center 
Attn:  STRNC-UP 
Natick, MA  01760-5017 

OFFICE  PHONE:  (508)-641-4122  DSN:  256-506^ 
WG:  Threat  Assessment 


Renee  Goldgraben 
The  MITRE  Corporation 
7525  Colshire  Drive 
MS  W936 

McLean, VA  22102 

OFFICE  PHONE:  (703) -883-6013 

WG:  Effectiveness  Methodologies 

DR  Gordon  J.  Goodwin 

US  Army  TRAC-FBH 

Attn:  ATRC-B 

Bldg  401B  —  TRAC 

Fort  Benjamin  Harrison, IN  46216 

OFFICE  PHONE:  (317) -543-6897  DSN:  699-6896 

WG:  Test  &  Evaluation 

Susan  Graves 

Coastal  Systems  Station 

Code  N043 

Panama  City,FL  32407-5000 
OFFICE  PHONE:  (904) -234-4138 
WG:  Threat  Assessment 
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Richard  S  Gray 
US  Army  MTMC 
Attn:  MTT-OA 
PO  Box  6276 

Newport  News.VA  23606-0276 
OFFICE  PHONE:  (804) -599-1650 
WG:  Test  &  Evaluation 

LtCol  Jimmy  R  Grussmeyer 
ODUSA  (OR) 

The  Pentagon,  Room  2E660 

Washington, DC  20310-0102 

OFFICE  PHONE:  ( 703 ) -697-0366  DSN:  227-0366 

WG:  Test  &  Evaluation 


Roland  R  Groover  Jr 
US  Army  TRAC-OAC 
ATTN:  ATRC-FSI 

Fort  Leavenworth.KS  66027-5200 

OFFICE  PHONE:  (913)-684-3533  DSN:  552-3533 

WG:  Threat  Assessment 


Ronald  A  Gustafson 

HQ  AFOTEC/CNP 

Kirtland  AFB.NM  87117-7001 

OFFICE  PHONE:  (505)-846-1844  DSN:  246-1844 

WG:  Test  &  Evaluation 


Mark  D  Haas 

TRW/BMO 

PO  Box  1310 

Building  520,  Rm  111 

San  Bernardino , CA  92402 

OFFICE  PHONE:  ( 714) -382-5307 

WG:  Future  Directions 

Michael  Hammes 
Naval  Sea  Systems  Command 
OFFICE  PHONE:  ( 703) -602-1 209 
WG:  Costing  Methodologies 


Capt  Daniel  F.  Harrington 
HQ  USMC 
Code  MA 

Washington, DC  20380-0001 

OFFICE  PHONE:  (703)-614-4165  DSN:  224-4165 

WG:  Effectiveness  Methodologies 


Alfred  G  Heinemann  III 
Global  Associates,  Ltd 
Suite  205 

2300  Clarendon  Blvd. 
Arlington, VA  22201 
OFFICE  PHONE:  (703) -351 -5660 
WG:  COEA  Management 


Charles  R  Hall  III 
The  MITRE  Corporation 
7525  Colshire  Drive 
McLean, VA  22102-3481 
OFFICE  PHONE:  (703) -883-6260 
WG :  COEA  Management 


David  L  Hammond 
ASD/XR 

Area  B,  Bldg  450,  Rm  B-105 
Wright-Patterson  AFB.OH  45433 
OFFICE  PHONE:  (513) -255-5288 
WG:  Costing  Methodologies 


Brinton  K.  Harrison 
HQDA 

DAMI-FIT 

The  Pentagon,  Rooom  2E453 
Washington, DC  20310-1088 

OFFICE  PHONE:  (703)-614~8121  DSN:  224-8121 
WG:  Threat  Assessment 

Richard  E  Helmuth 

Douglas  Aircraft  Company 

Mail  Code  35-95 

3855  Lakewood  Blvd 

Long  Beach, CA  90846 

OFFICE  PHONE:  (310)-593-7241 

WG:  Effectiveness  Methodologies 
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David  F  Hemingway 
BDM 

950  Explorer  Blvd 
Huntsville, AL  35806 
OFFICE  PHONE:  (205) -922-5282 
WG:  Effectiveness  Methodologies 


Mary  H.  Henry 

Training  Dev  &  Analysis  Div,  ODCST 

Attn:  ATTG-CR 

Fort  Monroe, VA  23651-5000 

OFFICE  PHONE:  ( 804) -728-5580  DSN:  680-5580 

WG:  Costing  Methodologies 


Robert  B  Hoffman 

General  Dynamics  Pomona  Div 

Operations  Res.  Section  6-134  (MZ  4-4) 

P0  Box  2507 
Pomona, CA  91769-2507 
OFFICE  PHONE:  (7 14) -868-4293 
WG:  Threat  Assessment 

Walter  W  Hollis 

DUSA  (OR) ,  Hq  Dept  of  the  Army 
ATTN:  SAOS(OR) 

Pentagon,  Room  2E660 
Washington, DC  20310-0102 

OFFICE  PHONE:  (703) -695-0083  DSN:  225-0083 
WG:  Speaker  or  Panelist 

LtCol  Kenneth  I  Hunt 

Defense  Systems  Management  College 

Attn:  FD-FM 

Fort  Belvoir.VA  22060-5426 

OFFICE  PHONE:  (703)-805-2451  DSN:  655-2851 

WG:  Costing  Methodologies 


Bobby  Jackson 
ANSER 
Suite  800 

1215  Jefferson  Davis  Hwy 
Arlington, VA  22202 
OFFICE  PHONE:  (703)-685-3379 
WG:  Costing  Methodologies 


DR  Jacqueline  R.  Henningsen 
HQ  SAC/XPA 

901  SAC  Blvd,  Suite  2B5 
Offutt  AFB.NE  68113-5330 

OFFICE  PHONE:  (402)-294-2355  DSN:  271-2355 
WG:  Synthesis  Group 


CDR  Stanford  H  Hlavka 

Office  of  the  Chief  of  Naval  Ops 

OP-815D 

Department  of  the  Navy 
Washington, DC  20350-2000 

OFFICE  PHONE:  (703)-697-8078  DSN:  227-0365 
WG:  COEA  Management 

Wilbur  C  Hogan  III 
US  Army  TRADOC 
ATTN:  ATAN-ZC 
Fort  Monroe, VA  23651-5000 

OFFICE  PHONE:  (804) -728-5803  DSN:  680-5803 
WG:  Costing  Methodologies 


C.  Frederick  Horr 
Hughes  Aircraft  Co 
Corporate  Mission  Analysis 
PO  Box  80028,  MS  C1/C147 
Los  Angeles, CA  90080 
OFFICE  PHONE:  (310)-568-7815 
WG:  COEA  Management 

Paul  R.  Ingholt 
DIA/DIAC 

Defense  Planning  Team,  GMI-IM 
Washington, DC  20340-6100 

OFFICE  PHONE:  (202) -373-3033  DSN:  243-3033 
WG:  Threat  Assessment 


Capt  Leslie  M  Jacobi 
OASN  (RD&A) 

Dept  of  the  Navy 
Pentagon,  Room  5E715 
Washington, DC  20350-1000 
OFFICE  PHONE:  ( 703) -614-4290 
WG:  COEA  Management 


207 


Attendance  at  MORS  Mini-Symposium:  COEA  .  .  .  (cont.) 


LtCol  Roger  A.  Jacobs 
OASD  ( PA&E ) 

The  Pentagon 
Washington, DC  20301 
OFFICE  PHONE:  ( 703 ) -697-0223 
WG:  Costing  Methodologies 


MAJ  James  R.  Johnson 
OASD (PA&E) (GPP) 

Lane  Forces  Division 

The  Pentagon,  Room  2B256 

Washington, DC  20301-1800 

OFFICE  PHONE:  (703)-697-7768  DSN:  227-7768 

WG:  Costing  Methodologies 

Robert  R  Jones 

David  Taylor  Research  Center 
Code  1210 

Washington, DC  20084-5000 

OFFICE  PHONE:  (301 )-227-4012  DSN:  287-4012 
WG:  Costing  Methodologies 


PROF  Kevin  P  Kelley 

Naval  War  College 

NSDM  Dept 

Newport, RI  02841 

OFFICE  PHONE:  (401 ) -841-3540 

WG:  Threat  Assessment 


MAJ  Toy  D  Kincer 
ESD/TGN 

Hanscom  AFB.MA  01731 

OFFICE  PHONE:  (617 ) -377-8478  DSN:  478-8478 
WG:  COEA  Management 


Jerry  A  Kotchka 
McDonnell  Douglas 
Missile  Systems  Company 
PO  Box  516  -  Mail  Code  1064605 
St.  Louis, MO  63166-0516 
OFFICE  PHONE:  (314)-232-2284 
WG:  Costing  Methodologies 


Joseph  C  Jenkins 
SIGMATECH 

4801  Bradford  Drive 
Huntsville, AL  35805-0797 
OFFICE  PHONE:  (205)-721-1188 
WG:  Future  Directions 


MAJ  Roy  T  Johnson 

HQ  AFOTEC/XPX 

Kirtland  AFB , NM  87117-7001 

OFFICE  PHONE:  (505)-846-5242  DSN:  246-5242 

WG:  Test  &  Evaluation 


COL  James  L  Kays 
US  Military  Academy 
Department  of  Systems  Engineering 
West  Point, NY  10996 

OFFICE  PHONE:  (914)-938-2701  DSN:  688-2701 
WG:  Synthesis  Group 


John  C  Key 

Coleman  Research  Corporation 
6820  Moquin  Drive 
Huntsville, AL  35806 
OFFICE  PHONE:  ( 205) -922-6000 
WG:  Effectiveness  Methodologies 


MAJ  James  C  Kolding 

US  Army  TRADOC  Analysis  Center-WSMR 

ATTN:  ATRC-ZD 

Bldg  1400 

White  Sands  Missile  Range, NM  88002-5502 
OFFICE  PHONE:  (505) -678-7690  DSN:  258-7690 
WG:  Effectiveness  Methodologies 

LtCol  Robert  R  Koury 

US  Army  Concepts  Analysis  Agency 

8120  Woodmont  Avenue 

Bethesda,MD  20814 

OFFICE  PHONE:  ( 301 ) -295-1546 

WG:  Costing  Methodologies 
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Glen  A  Kraft 
HQ  AFOTEC/OAW 
Kirtland  AFB,NM  87117 
OFFICE  PHONE:  (505) -886-4203 
WG:  Test  &  Evaluation 


Ronald  P  Kramer 

Naval  Sea  Systems  Command 

Code  50143 

Washington, DC  20362-5101 
OFFICE  PHONE:  (202)-692-8252 
WG:  Threat  Assessment 


Dale  E  Kristof 
ASD/XR 

Area  B,  Bldg  450,  Rm  B-118 
Wright-Patterson  AFB,0H  45433 
OFFICE  PHONE:  (513)-255-5880 
WG:  Threat  Assessment 


Franklin  K.  Ladner 

LTV  Aerospace  and  Defense 

Missiles  Division,  MS  WT-52 

PO  Box  650003 

Dallas, TX  75265-0003 

OFFICE  PHONE:  (214)-266-9233 

WG:  Threat  Assessment 


Glenn  F  Lamartin 
0USD(A)/TWP 
Pentagon,  Room  3D1048 
Washington, DC  20301 

OFFICE  PHONE:  (703) -695-6394  DSN:  225-6394 
WG:  Effectiveness  Methodologies 


Mary  A.  Lambert 

The  MITRE  Corporation 

202  Burlington  Road 

Bedford, MA  01730 

OFFICE  PHONE:  (617)-271-7980 

WG:  Effectiveness  Methodologies 


MAJ  Paul  J  Landwehrle 
HQ  TAC/XP-JSG 
Langley  AFB.VA  23665 

OFFICE  PHONE:  (804)-764-5717  DSN:  574-5717 
WG :  COEA  Management 


Joann  H  Langston 
OASA(RDA) 

ATTN:  SARD-CA 

5109  Leesburg  Pike,  Suite  302 

Falls  Church, VA  22041-3201 

OFFICE  PHONE:  (703)-756-7576  DSN:  289-7576 

WG:  Future  Directions 

Jeffery  H  Leech 
HQ  ASD/ENSS 

Wright-Patterson  AFB,OH  45433-6503 
OFFICE  PHONE:  (513) -476-4824 
WG:  Effectiveness  Methodologies 


Capt  Wallace  Langbehn 

HQ  TAC/XP-JSG 

Langley  AFB , VA  23665-5520 

OFFICE  PHONE:  (804) -764-5460  DSN:  574-5460 
WG:  Effectiveness  Methodologies 


Richard  R  Ledesma 
OUSD(A) ,  DDDRE(T&E) 

The  Pentagon,  Room  3E1075 
Washington, DC  20301-3110 

OFFICE  PHONE:  (703) -695-4421  DSN:  225-4421 
WG:  Test  &  Evaluation 


Joseph  R.  Lehman 
PRC 

2750  Killarney  Drive,  Suite  200 
Woodbridge.VA  22192 
OFFICE  PHONE:  ( 703) -690-1993 
WG:  COEA  Management 
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Edward  J  Leinfelder 

LTV  Missiles  &  Electronics  Group 

MS  WT  52 

PO  Box  650003 

Dallas, TX  75265-0003 

OFFICE  PHONE:  (214)-266-9201 

WG:  Threat  Assessment 

James  E.  Lewis 

US  Total  Army  Personnel  Command 

Attn:  TAPC-PI-MPA 

200  Stovall  Street 

Alexandria, VA  22332-1345 

OFFICE  PHONE:  ( 703 ) -325-2090  DSN:  221-2090 

WG:  Costing  Methodologies 

MAJ  Timothy  G  Loftis 
ASD/FMCC 

Wright-Patterson  AFB , OH  45433 

OFFICE  PHONE:  (513)-255-6347  DSN:  785-6347 

WG:  Costing  Methodologies 


DR  William  G  Lese  Jr 
0ASD  ( PA&E ) 

The  Pentagon,  Room  2B256 

Washington, DC  20301-1800 

OFFICE  PHONE:  ( 703 ) -695-0881  DSN:  225-0881 

WG:  Test  &  Evaluation 


David  A.  Loental 

US  Army  Engineer  School 

ATTN:  ATSE-CDC-M 

Fort  Leonard  Wood, MO  65473 

OFFICE  PHONE:  ( 314) -563-5449  DSN:  676-5449 

WG:  Effectiveness  Methodologies 


Thomas  P  Lubinski 
ASD/XRH 

Wright-Patterson  AFB, OH  45433-6503 
OFFICE  PHONE:  (513)-255-3168 
WG:  Effectiveness  Methodologies 


DR  Bruce  A  MacDonald 
The  MITRE  Corporation 
MS  W  272 

7525  Colshire  Dr 
McLean, VA  22102 
OFFICE  PHONE:  (703) -883-7855 
WG:  Test  &  Evaluation 

Jeffrey  A  Manickas 
Naval  Undersea  Warfare  Center 
Division  Newport 
Newport, RI  02841-5047 

OFFICE  PHONE:  (401) -841-4299  DSN:  948-4299 
WG:  Costing  Methodologies 


DR  Gerald  R  McNichols 
Management  Consulting  &  Research,  Inc 
5113  Leesburg  Pike 
Suite  509 

Falls  Church, VA  22041 
OFFICE  PHONE:  ( 703) -820-4600 
WG:  Synthesis  Group 


MAJ  David  S.  Malcolm 
Marine  Corps  Systems  Command 
Analysis  Support  Branch 
Code  PSA 

Quantico,VA  22134-5080 
OFFICE  PHONE:  (703) -640-2420 
WG :  COEA  Management 

James  T.  McCormick 
SAIC 

8619  Westwood  Center  Drive 
Suite  608 
Vienna, VA  22182 
OFFICE  PHONE:  (703)-749-5603 
WG:  Test  &  Evaluation 

LtCol  L.  James  Merryman 
Joint  Staff,  J-8 
The  Pentagon,  Room  1D964 
Washington, DC  20318-8000 

OFFICE  PHONE:  (703)-614-3681  DSN:  224-3681 
WG:  Future  Directions 
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Vincent  P  Merz 

VEDA,  Inc 

PAXR  Division 

300  Exploration 

Lexington  Park,MD  20653 

OFFICE  PHONE:  ( 301 ) -862-2100 

WG:  Test  &  Evaluation 

Robert  J  Meyer 

Naval  Air  Warfare  Center 

Code  3186 

Weapons  Division 

Ridgecrest, CA  93555-6001 

OFFICE  PHONE:  (619 ) -939-6686  DSN:  437-6686 

WG:  Effectiveness  Methodologies 

Donald  E  Mixon 

The  MITRE  Corporation 

7525  Colshire  Drive 

McLean, VA  22102 

OFFICE  PHONE:  (703) -883-6599 

WG:  Costing  Methodologies 


William  M.  Mulholland 
McDonnell  Aircraft  Company 
MC  0642233 
PO  Box  516 

St.  Louis, M0  63166-0516 
OFFICE  PHONE:  (314) -233-4433 
WG:  Threat  Assessment 

Mark  W  Murray 

HQ  TRADOC 

Attn:  ATAN-AP 

Fort  Monroe, VA  23651-5000 

OFFICE  PHONE:  (804)-728-5834  DSN:  680-5834 

WG:  Future  Directions 


Fiona  B.  Nave 

Naval  Air  Warfare  Center 

Code  36251 

Ridgecrest, CA  93555 

OFFICE  PHONE:  (619 ) -939-1927 

WG:  Future  Directions 


DR  James  J  Metzger 
ODCSOPS,  HQ  Dept  of  the  Army 
ATTN : DAMO-ZDS 
Washington, DC  20310-0410 

OFFICE  PHONE:  (703)-695-2451  DSN:  225-2451 
WG:  Effectiveness  Methodologies 


Capt  Steven  A.  Minarik 
Marine  Corps  Systems  Command 
Analysis  Support  Branch 
Code  PSA 

Quantico.VA  22134-5080 
OFFICE  PHONE:  (703) -640-2420 
WG:  Test  &  Evaluation 

James  S  Moore 
Coastal  Systems  Station 
Code  N043 

Panama  City.FL  32407-4505 
OFFICE  PHONE: 

WG :  COEA  Management 


Richard  P  Munro 
SAIC 

MS  1-13-2 

1710  Goodridge  Dr 

McLean, VA  22102 

OFFICE  PHONE:  (703) -827-4764 

WG:  COEA  Management 

Robert  M.  Nash 

Marine  Corps  Systems  Command 
Analysis  Support  Branch 
Code  PSA 

Quantico.VA  22134-5080 
OFFICE  PHONE:  (703) -640-2420 
WG:  Costing  Methodologies 

MAJ  Tom  Novak 
ESD/TGN 

Hanscom  AFB ,MA  01731 
OFFICE  PHONE:  (617)-377-8920 
WG:  COEA  Management 
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DR  Daniel  A  Nussbaum 

Naval  Center  For  Cost  Analysis 

Rm  4A538 

NCA-4 

Washington, DC  20350-1100 

OFFICE  PHONE:  (703)-746-2327  DSN:  286-2327 

WG:  Costing  Methodologies 

Kim  M.  Oliver 
HQ  USAES 

Attn:  ATSE-CDC-A 

Fort  Leonard  Wood, MO  65473-6620 

OFFICE  PHONE:  (314)-563-7887  DSN:  676-7887 

WG:  COEA  Management 


Harold  C  Pasini 
US  Army  OEC 
Oark  Center  IV 
4501  Ford  Avenue 
Alexandria, VA  22302-1458 

OFFICE  PHONE:  (703)-756-2294  DSN:  289-2294 
WG:  Test  &  Evaluation 

William  C  Peters 

ANSER 

Suite  800 

1215  Jefferson  Davis  Hwy 
Ar lington, VA  22202 
OFFICE  PHONE:  (703) -685-3350 
WG:  Test  &  Evaluation 

Kenneth  W  Potempa 
HSD/XRX 

Brooks  AFB , TX  78235 

OFFICE  PHONE:  (512)-536-2424  DSN:  240-2424 
WG:  Costing  Methodologies 


PROF  Peter  Purdue 
Naval  Postgraduate  School 
Department  of  Operations  Research 
Code  OR 

Monterey, CA  93943 

OFFICE  PHONE:  (408)-646-2381  DSN:  878-2381 
WG:  Effectiveness  Methodologies 


CAPT  Michael  S.  O' Hearn 
Operational  Test  and  Evaluation  Force 
ACOS,  OT&E  Support  (30  Div) 

Norfolk, VA  23511-5225 

OFFICE  PHONE:  (804)-444-5063  DSN:  564-5063 
WG:  Test  &  Evaluation 


Anita  A  Ontiveros 

US  Army  Air  Defense  Artillery  School 

Attn:  ATSA-CDC-S 

Fort  Bliss, TX  79916-0002 

OFFICE  PHONE:  (915)-568-2233  DSN:  978-2233 

WG:  Future  Directions 


DR  Ralph  W.  Passarelli 
Center  for  Naval  Analyses 
4401  Ford  Avenue 
Alexandria, VA  22302-0268 
OFFICE  PHONE:  (703)-614-4290 
WG:  Future  Directions 


Robert  L.  Pierce 

Lockheed  Aeronautical  Systems  Company 

86  South  Cobb  Drive 

Marietta, GA  30063 

OFFICE  PHONE:  (404) -494-1966 

WG:  Effectiveness  Methodologies 


Karen  W  Pullen 

The  MITRE  Corportion 

202  Burlington  Road 

Bedford, MA  01730 

OFFICE  PHONE:  (617) -271-8475 

WG:  Threat  Assessment 


DR  Herbert  C  Puscheck 

OASD  (PA&E) DADS/GPP 

Pentagon,  Room  2E330 

Washington, DC  20301-1800 

OFFICE  PHONE:  (703) -695-0528  DSN:  225-052S 

WG:  Speaker  or  Panelist 


212 


Attendance  at  MORS  Mini-Symposium:  COEA  .  .  .  (cont.) 


Carrie  F.  Quesnell 
Naval  Air  Warfare  Center 
Code  36251 
Ridgecrest ,CA  93555 

OFFICE  PHONE:  (619)-939-3616  DSN:  437-1927 
WG:  Future  Directions 


MajGen  Joseph  W  Ralston 
HQ  USAF/XOR 
Pentagon,  Room  4E1021 
Washington, DC  20330 

OFFICE  PHONE:  (703)-614-8344  DSN:  224-8344 
WG:  Speaker  or  Panelist 


PROF  Raymond  W  Reig 

Defense  Systems  Management  College 

FD-TE 

Fort  Belvoir.VA  22060-5426 

OFFICE  PHONE:  ( 703 ) -805-2887  DSN:  354-2887 

WG:  Test  &  Evaluation 


MAJ  Allan  M  Resnick 
US  Army  TRAC-SAC 
ATTN:  ATRC-SA 
Fort  Leavenworth, KS  66027 
OFFICE  PHONE:  (913) -684-5427 
WG:  Threat  Assessment 


Alfred  S  Rhode  FS 
George  Mason  University 
OFFICE  PHONE:  (301)-217-7443 
WG:  Synthesis  Group 


Lance  M.  Roark 
OASD  (PA&E) 

The  Pentagon,  Room  2D322 
Washington, DC  20301-0180 
OFFICE  PHONE:  (703 >-693-7816 
WG:  Costing  Methodologies 


John  B  Rogers 

Coleman  Research  Corporation 
6820  Moquin  Drive 
Huntsville ,AL  35806 
OFFICE  PHONE:  (205) -922-6000 
WG:  Effectiveness  Methodologies 


Royce  H  Reiss 
AFCSA/SAG 

The  Pentagon,  Room  1D384 
Washington, DC  20330-5420 

OFFICE  PHONE:  (703)-695-5552  DSN:  255-5222 
WG:  Future  Directions 


Roy  F  Reynolds 
US  Army  TRAC-WSMR 
Close  Combat  Directorate 
Bldg  1400 

White  Sands  Missile  Range, NM  88002-5502 
OFFICE  PHONE:  (505) -678-4300  DSN:  258-430C 
WG:  Future  Directions 

DR  Daniel  T  Risser 

Dynamics  Research  Corporation 

60  Concord  St 

Wilmington, MA  01887 

OFFICE  PHONE:  (508) -658-6100 

WG:  Future  Directions 


Allen  D  Roe 
Veda,  Inc 

5200  Springfield  Pike,  //200 
Beavercreek, OH  45431-1255 
OFFICE  PHONE:  (513) -253-4770 
WG:  Costing  Methodologies 


Ronald  E.  Rogers 

The  Analytic  Sciences  Corporation  (TASC) 

2555  University  Blvd 

Fairborn, OH  45324 

OFFICE  PHONE:  (513)-426-1040 

WG:  Test  &  Evaluation 
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CDR  Henry  J  Rohling 

US  Special  Operations  Command 

Attn:  Code  S0J3-RC 

MacDill  AFB.FL  33608-6001 

OFFICE  PHONE:  (813 ) -830-2015  DSN:  968-2015 

WG:  Test  &  Evaluation 


LtCol  Jeffrey  R  Sackett 
HQ  TAC/XP-JSG 
4525  CAS/JSG 
Langley  AFB , VA  23665 

OFFICE  PHONE:  (804)-764-5754  DSN:  574-5754 
WG:  COEA  Management 


DR  Patricia  A.  Sanders 
OASD  ( PA&E ) 

Pentagon,  Room  2B256 

Washington, DC  20301-1800 

OFFICE  PHONE:  (703)-697-3521  DSN:  227-3521 

WG:  Test  &  Evaluation 


DR  John  Y  Schrader 
The  RAND  Corporation 
2100  M.  Street,  NW 
Washington, DC  20037-1270 
OFFICE  PHONE:  (202 ) -296-5000 
WG:  Threat  Assessment 


Capt  Thomas  J  Schwartz 
US  Army  Armor  School 
Attn:  ATSB-CDC 
Fort  Knox , KY  40121 

OFFICE  PHONE:  (502) -624-3648  DSN:  464-3648 
WG:  Effectiveness  Methodologies 


DR  Ernest  A.  Seglie 
OSD  (DOTE) 

Pentagon,  Room  3E318 
Washington, DC  20301-1700 

OFFICE  PHONE:  (703)-697-7247  DSN:  227-7247 
WG:  Speaker  or  Panelist 


Patricia  H  Rossmaier 
Naval  Surface  Warfare  Center 
Code  Gil 

Dahlgren.VA  22448 

OFFICE  PHONE:  (703) -663-8851  DSN:  294-8851 
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.  (cont.) 
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WG:  Effectiveness  Methodologies 

Clayton  J  Thomas  FS 
HQ  USAF/AFSAA 
Pentagon,  Room  1E386 
Washington, DC  20330-5420 

OFFICE  PHONE:  (703)-697-4300  DSN:  227-4300 
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WG:  Effectiveness  Methodologies 

Eugene  P  Visco  FS 
US  Army  MISMA,  Attn:  SFU-MIS 
Crystal  Square  2,  #808 
1725  Jefferson  Davis  Hwy 
Arlington, VA  22202 

OFFICE  PHONE:  (703)-746-8071  DSN:  286-8071 
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PO  Box  30 

State  College, PA  16804 
OFFICE  PHONE:  (814)-863-4102 
WG:  Test  &  Evaluation 

MAJ  Steven  L  Wingfield 
HQ  TAC/DRAS 

Langley  AFB , VA  23665-5575 

OFFICE  PHONE:  (804) -764-7066  DSN:  574-7066 
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